Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-11590

[C++] Move CSV background generator to IO thread pool

    XMLWordPrintableJSON

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 4.0.0
    • C++

    Description

      The current readahead accomplishes its task by using an async reentrant readahead.  We get away with this today because the background generator is async reentrant because it runs on a thread pool of size 1 and so the underlying thread pool can only iterate the underlying iterator synchronously.

      However, in order to move to the I/O thread pool we need a readahead operator that is does not pull reentrantly.

      This can be done either by submitting the next task as soon as the previous one is done or by creating a "serial concurrency" executor that wraps an underlying executor.  Both approaches will need a queue so this can be done after ARROW-11588

      Attachments

        Issue Links

          Activity

            People

              westonpace Weston Pace
              westonpace Weston Pace
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 7.5h
                  7.5h