Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-3762

[C++] Parquet arrow::Table reads error when overflowing capacity of BinaryArray

    Details

      Description

      When reading a parquet file with binary data > 2 GiB, we get an ArrowIOError due to it not creating chunked arrays. Reading each row group individually and then concatenating the tables works, however.

       

      import pandas as pd
      import pyarrow as pa
      import pyarrow.parquet as pq
      
      
      x = pa.array(list('1' * 2**30))
      
      demo = 'demo.parquet'
      
      
      def scenario():
          t = pa.Table.from_arrays([x], ['x'])
          writer = pq.ParquetWriter(demo, t.schema)
          for i in range(2):
              writer.write_table(t)
          writer.close()
      
          pf = pq.ParquetFile(demo)
      
          # pyarrow.lib.ArrowIOError: Arrow error: Invalid: BinaryArray cannot contain more than 2147483646 bytes, have 2147483647
          t2 = pf.read()
      
          # Works, but note, there are 32 row groups, not 2 as suggested by:
          # https://arrow.apache.org/docs/python/parquet.html#finer-grained-reading-and-writing
          tables = [pf.read_row_group(i) for i in range(pf.num_row_groups)]
          t3 = pa.concat_tables(tables)
      
      scenario()
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bkietz Benjamin Kietzman
                Reporter:
                LeftScreenCorner Chris Ellison
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 8h 10m
                  8h 10m