Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-7001

[C++] Develop threading APIs to accommodate nested parallelism

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 5.0.0
    • C++

    Description

      Tasks invoked in parallel may be able to submit their own subtasks, which in OpenMP and TBB documentation is often called "nested parallelism".

      If a task blocks on the completion of subtasks, then outright deadlocks are possible – running tasks are all blocking on their subtasks, but the thread pool will not schedule any further tasks.

      I suggest that such code have a way to indicate to the thread pool (if one is passed in) that it is blocking on the completion of other tasks so that further tasks can be run while the task waits for its child tasks to complete. One possible way to do this is to have a floating "soft limit" for concurrent tasks that can be incremented when tasks are waiting.

      So if we normally allow 8 concurrent tasks, then this can be temporarily increased for each "suspended" task. Preferably we would provide some way for the dependent task group to "awaken" the suspended task so that it does not have to do any work while waiting for the task group to finish

      Note this feature can also be used in tasks that are waiting for IO calls

      Attachments

        Issue Links

          Activity

            People

              westonpace Weston Pace
              wesm Wes McKinney
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 41h 40m
                  41h 40m