I wonder if we need a way to store the results of a task until they are fetched? I'm thinking of something like update-by-query which would be a task because it is long running, cancelable, etc. But it wants to return counts of how many documents it updated and things like that. Maybe just write them to an index? Maybe with a ttl?

Nice remark of @nik9000 about long task results availability after its end, as it is already done for snapshots.
But I'd prefer a TTL or/and a dedicated query to drop the result instead of only drop on fetch, which could be problematic in my opinion.

But I'd prefer a TTL or/and a dedicated query to drop the result instead of only drop on fetch, which could be problematic in my opinion.

Yeah - drop on fetch would be rough.

Not all tasks will want to do this but I think some would like it.

@nik9000 is the goal to make results available after the task finished?

@nik9000 is the goal to make results available after the task finished?

Yeah. In the case of update-by-query it'd be just to make the status available. The most "convenient" way to do it seems like write it to an index with a ttl - but I think I'm just stuck on that idea because it came to me. The point is that after the task is done you'll want to see what its results were for some period of time. You'd want some place you could fetch the results by task id, some way to clear out results when you've finished with them, some way for them to clear themselves out if you don't read them back soon enough.

I don't think it needs to come at iteration 1, but at some point it'd be nice.

Look at delete-by-query, it makes some effort to build a nice results object. Once it becomes a "task" it'll have nothing to do with the fancy result object.

Another thing that might be useful is to make an API that'd block until the task was finished and return the result of it. Or just fetch the result if it was already finished. This'd be super useful in general but kind of required for the REST tests because they don't have loops and things.

I think traditionally we do that in two places - 1) log files for per-operation level and 2) in stats as combined metric. I can see how we might want to have a third way, but I think the biggest question here is lifecycle of this result. Persistence (even temporary persistence) of results is very unclear to me unless the result is associated with some persistent object (such as snapshot). So, I would rather make it an option to block and get result if you are interested in the result.

So, I would rather make it an option to block and get result if you are interested in the result

I don't know if that'll be enough in the end though. Imagine the delete-by-query operation that takes 30 minutes too complete. Its too long for any blocking to be reliable - all kinds of http equipment will time you out after 5 minutes and something is bound to sneak in and get you a connection reset by peer.

So you'd have to build in a retry to the blocking. But if results aren't persisted, at least for a little while, then there is always the possibility that the job will finish between one request timing out and the next one starting. A low possibility but an icky one.

Something like a TTL on the result with explicit commands to read the result and delete it would work. These results wouldn't be huge documents so we could probably keep them in memory, certainly if they were serialized xContent or implemented Accountable or something.

Its complicated but I can't think of how else to report on tasks that are "do a thing" rather than "make a thing".

+1

Another user of task mgmt: the forced merge API

Another user of task mgmt: the forced merge API

I wonder if we should add a list of users to the top, like, below the requirements. I'm happy to work on retrofitting some of our long running requests to make their status more fetch-able and to make them more cancel-able but we should should make a list/tag the old issues.

Will it be possible to suspend/resume tasks by API? For perpetual tasks? Or to schedule tasks by a cron-like specification? It seems not, since the task design discusses TransportAction only, which means the lifetime of a task is "one-shot", corresponding to a request/response roundtrip of an action executed by a user?

@jprante at the moment we are targeting use cases where tasks have clear start, stop and finite running time. We might extend it to perpetual tasks in the future, but this is not on the immediate road map.

Will it be possible to suspend/resume tasks by API?

I suspect that'll be opt-in in the same way that cancel will be opt-in. Reindex will probably opt in because it'll want to have API controlled throttling. So you could set the throttle to 0 and it'd just stop. The bulk request powering it would timeout pretty soon, making the whole thing fail though.

We will has the internal result of a long-term-running task? For example:
The query will retrieve total 100 shards on 10 elastic node, each node has 10 shards. We can get the result immediately when the first shard is done. Then the second shard is done, we reduce the result to the former, send notify to the API, so the user can update their view. When all the shards are done, send notify that the task is done.
So we can execute this long-term-running task in background with less thread one shard by another, release the CPU resources for the high-priority task. And the user can update the views of long-term-running task frequently, got a better user experience.

@clintongormley good idea, I have added it to the description

@imotov, is this done now? I think we've decided not to do the "task can survive restart" thing, right?

@clintongormley @imotov #15975 it was closed in favor of this ticket, so can we retrieve information about _forcemerge?

@oleg-andreyev thanks! I have reopened #15975.