Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-15968

[C++] Update AsyncGenerator semantics to emit a terminal item only after all outstanding futures have completed

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 8.0.0
    • C++

    Description

      Currently, when an error occurs, we emit the error immediately. Once an error is emitted from the generator we typically discard it and all related state. Unfortunately, in readahead situations, there may be outstanding tasks, and these may be referencing that related state.

      Currently, we avoid errors in this situation through the copious use of shared_ptr. This has led to excess complexity and potential performance issues.

      Instead, we can update the rules for async generators to only emit a terminal item once all outstanding tasks have completed. At this point we should be able to safely tear down all of the state. Furthermore, if we can ensure consumers drain an async generator before they destroy it (this can be done in a follow-up issue), we can get rid of a lot of our shared_ptr and simplify some of the complexity of the async reader code.

      Attachments

        Issue Links

          Activity

            People

              westonpace Weston Pace
              westonpace Weston Pace
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 4h 40m
                  4h 40m