@jon-wei thanks for writing this up, some questions:

• Do we get query prioritization support? Since the query-serving resources are a shared pool among all the tasks, we'd want to make sure that important queries can take precedence over heavier ones.

• What does the query endpoint look like? Will it be shared among all tasks similar to the chat handler (rather than using a different port for each task)? Something like POST http://{bigIndexerHost}:{bigIndexerPort}/druid/v2/{taskId}

• Regarding globalIngestionHeapLimitBytes:

When the sum of ingestion memory usage and merging memory usage across all tasks reaches globalIngestionHeapLimitBytes, the BigIndexer will trigger a global persist, causing each managed task to persist its in-heap segment data.

The BigIndexer will also have a mode where the global limit is divided evenly across the number of worker slots.

I think having a global heap limit configuration instead of just using the per-task maxBytesInMemory is great and would allow us to optimize our usage of the available memory. I don't really understand why we need to have a mode that divides the heap into fixed portions; what is the benefit of this? To me this feels like a throwback to the current model where every worker is allocated the same memory, whether it needs it or not, and large tasks suffer when run together with a bunch of small tasks. The main advantage I can see for this is that the global spill will not trigger in this case, which prevents unnecessary spills when you have a bunch of small tasks running with some big tasks that would otherwise keep triggering the global spill. As another option to consider, what do you think about having a configuration parameter that controls the minimum bytes in memory that needs to be reached before a sink is eligible to be spilled during a global spill? The default value could be something like globalIngestionHeapLimitBytes / druid.worker.capacity + some accounting for merging memory requirements. This would give the flexibility of heavy tasks being able to utilize more than its 'equal share' of memory, while preventing unnecessary fragmentation for lightweight tasks.

• How do shared lookups work? Are all lookups loaded into memory when the BigIndexer starts, regardless of whether it is running tasks and if those tasks are query-serving? Will a BigIndexer release lookups that haven't been used in a while?
• Can the connection/thread pool for chat handlers be separate from the one used for serving queries? For supervised tasks, the supervisor tends to flip out if the task stops responding to control/status requests, and if the pool is shared, I could see some annoying issues where heavy query load to one set of tasks causes the supervisor to kill completely unrelated tasks for being unresponsive.

I think we should just call it an indexer process.

@dclim

Do we get query prioritization support? Since the query-serving resources are a shared pool among all the tasks, we'd want to make sure that important queries can take precedence over heavier ones.

Yes, this should be supported.

What does the query endpoint look like? Will it be shared among all tasks similar to the chat handler (rather than using a different port for each task)? Something like POST http://{bigIndexerHost}:{bigIndexerPort}/druid/v2/{taskId}

I'm thinking it should be shared across all tasks, but the endpoint would just be http://{bigIndexerHost}:{bigIndexerPort}/druid/v2 without a taskId

How do shared lookups work? Are all lookups loaded into memory when the BigIndexer starts, regardless of whether it is running tasks and if those tasks are query-serving? Will a BigIndexer release lookups that haven't been used in a while?

For simplicity in the initial implementation, I'm thinking all lookups should be loaded into memory when the indexer starts. (Re: query-serving, I believe lookups can also be used during ingestion, via transform specs->lookup expressions)

Can the connection/thread pool for chat handlers be separate from the one used for serving queries? For supervised tasks, the supervisor tends to flip out if the task stops responding to control/status requests, and if the pool is shared, I could see some annoying issues where heavy query load to one set of tasks causes the supervisor to kill completely unrelated tasks for being unresponsive.

That sounds like a good idea, I'll look into that.

To me this feels like a throwback to the current model where every worker is allocated the same memory, whether it needs it or not, and large tasks suffer when run together with a bunch of small tasks. The main advantage I can see for this is that the global spill will not trigger in this case, which prevents unnecessary spills when you have a bunch of small tasks running with some big tasks that would otherwise keep triggering the global spill.

Hm, it was intended as a throwback mode, I'll reconsider whether that's useful to include.

As another option to consider, what do you think about having a configuration parameter that controls the minimum bytes in memory that needs to be reached before a sink is eligible to be spilled during a global spill? The default value could be something like globalIngestionHeapLimitBytes / druid.worker.capacity + some accounting for merging memory requirements. This would give the flexibility of heavy tasks being able to utilize more than its 'equal share' of memory, while preventing unnecessary fragmentation for lightweight tasks.

This sounds useful, I will look into this as well.

@fjy

I think we should just call it an indexer process.

That sounds good.

@jon-wei thanks! Two other things that I was thinking about that might be interesting, or might be out of scope for this proposal:

• A GC watchdog that tries to save an indexer that's going to die unnecessarily - this has been alleviated somewhat by using maxBytesInMemory instead of maxRowsInMemory, but I think can still be an issue because query heap usage is not strictly bounded and unexpected query load could cause OOM issues (and this effect would be compounded if you have multiple tasks running in the same JVM), but basically: indexing tasks often die because of heap exhaustion unnecessarily - meaning that the data in the heap didn't need to be there, and if a spill would have been triggered, the indexing task could continue on. I'm thinking of some kind of watchdog thread that uses GarbageCollectorMXBean similar to JvmMonitor and if a threshold of number of GCs or GC time in a given period is hit, the watchdog will force a global spill to try to save the indexer.

• Some intelligence around task assignment that's better than random - an easy win here would be to weight known resource-light tasks (Hadoop indexing task, parallel master task?) lesser than other tasks. We might also want to consider query-serving realtime tasks separately from batch jobs and optimize for those (distributing them evenly across the indexers so that they're not all running on the same machine and sharing processing threads/buffers while the buffers are unused on other machines because they're running all batch jobs) - I think being able to respond to realtime queries quickly is typically more important than ingestion throughput. Later on, it would be pretty interesting to look at historical task report information to try to understand what are the heavy datasources (in terms of both query and ingestion volume) and what are the light ones and further optimize using this information.

Otherwise, I'm +1 on this proposal. Assuming that the new indexer will have enable/disable routes that toggle whether it will receive new task assignments similar to the MM for supporting rolling updates.

@dclim Great ideas, I think they make sense as follow-on features, I've added them to the Future Work section

@jon-wei nice proposal! Thank you for writing it up. The proposal looks good to me overall.

I just have a couple of questions.

• When a merge notice is received, the Indexer will attempt to record two memory allocations in its memory usage tracking state:
  • a fixed amount of heap. This can scale based on some proportion of globalIngestionHeapLimitBytes.
  • a fixed amount of direct memory, needed for decompression buffers and dictionary conversions. This can scale based on the size in bytes of the fuzzy +1 factor in the familiar (druid.processing.numThreads + druid.processing.numMergeBuffers + 1) * druid.processing.buffer.sizeBytes formula.

This sounds like each task needs to reserve some memory up front. How does the indexer schedule the memory reservation request?

Points raised in "motivation" are valid, so +1 for the proposal.

however...

A new process type, BigIndexer (tentative name), will be added. This new process type is an alternative to the MiddleManager that runs all tasks in the single BigIndexer JVM instead of forking new processes.

It looks like a MiddleManager that runs tasks in its own jvm process instead of spawning new processes. When I look at it from that perspective, it seems more logical to add a configuration parameter to existing MiddleManager node type that says whether or not to run task processes in same jvm process.

When I look at it from that perspective, it seems more logical to add a configuration parameter to existing MiddleManager node type that says whether or not to run task processes in same jvm process.

@himanshug I found that it was cleaner to define a new node type, since there are aspects of the new process that diverge from the existing MM's behavior.

For example, with lookups, MM process doesn't need to load them but the new indexer does. In DruidNodeDiscoveryProvider, I would otherwise have to associate the MiddleManager node type with LookupNodeService and doing that kind of control across the board with a shared node type seems complicated.