Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Won't Fix
-
1.0.1
-
None
Description
Currently ThreadedTaskGroup::Finish locks the current thread waiting for worker threads to execute tasks. Instead it could pop tasks from the queue and execute them, using the Finish-ing thread as a worker. This would enable basic nested parallelism cases using TaskGroup::MakeSubGroup() without danger of accumulating a thread deadlock.
For example in the case of reading multiple parquet files we would like to parallelize both across files to read and across columns within each file. We could support this basic nested parallelism by rewriting ParquetFileReader accept any TaskGroup across which to scatter its column reading tasks (rather than instantiating its own ThreadPool based on a boolean flag). Then file reading tasks could be scattered across a ThreadedTaskGroup, each of these creating a subgroup which runs all column reading tasks.
However the above would currently deadlock for reading (# files) * (# columns) >= (# threads), since every task of the root TaskGroup will be locked by its subgroup's call to Finish. In order to use TaskGroup::MakeSubGroup for basic nested parallelism, the Finish-ing thread must perform work in addition to checking for group completion.