Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-13441

[CSV] Streaming reader conversion should skip empty blocks

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 5.0.0
    • 6.0.0
    • C++

    Description

      The csv streaming reader hardens the schema after the first block is processed. However if the first block does not have any rows then the schema will be hardened with all columns as NAType. This is made worse with the skip_rows_after_names options which will create empty batches until the specified number of rows are skipped.

      Attachments

        Issue Links

          Activity

            People

              neworld Nate Clark
              neworld Nate Clark
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 5.5h
                  5.5h