Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
Description
Tasks invoked in parallel may be able to submit their own subtasks, which in OpenMP and TBB documentation is often called "nested parallelism".
If a task blocks on the completion of subtasks, then outright deadlocks are possible – running tasks are all blocking on their subtasks, but the thread pool will not schedule any further tasks.
I suggest that such code have a way to indicate to the thread pool (if one is passed in) that it is blocking on the completion of other tasks so that further tasks can be run while the task waits for its child tasks to complete. One possible way to do this is to have a floating "soft limit" for concurrent tasks that can be incremented when tasks are waiting.
So if we normally allow 8 concurrent tasks, then this can be temporarily increased for each "suspended" task. Preferably we would provide some way for the dependent task group to "awaken" the suspended task so that it does not have to do any work while waiting for the task group to finish
Note this feature can also be used in tasks that are waiting for IO calls
Attachments
Issue Links
- is related to
-
ARROW-6964 [C++][Dataset] Expose a nested parallel option for Scanner::ToTable
-
- Resolved
-
-
-
- Resolved
-
- links to
