Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-2462

[C++] Segfault when writing a parquet table containing a dictionary column from Record Batch Stream

    XMLWordPrintableJSON

    Details

    • Flags:
      Patch

      Description

      Discovered this through using pyarrow and dealing with RecordBatch Streams and parquet. The issue can be replicated as follows:

      import pyarrow as pa
      import pyarrow.parquet as pq
      
      # create record batch with 1 dictionary column
      indices = pa.array([1,0,1,1,0])
      dictionary = pa.array(['Foo', 'Bar'])
      dict_array = pa.DictionaryArray.from_arrays(indices, dictionary)
      rb = pa.RecordBatch.from_arrays( [ dict_array ], [ 'd0' ] )
      
      # write out using RecordBatchStreamWriter
      sink = pa.BufferOutputStream()
      writer = pa.RecordBatchStreamWriter(sink, rb.schema)
      writer.write_batch(rb)
      writer.close()
      buf = sink.get_result()
      
      # read in and try to write parquet table
      reader = pa.open_stream(buf)
      tbl = reader.read_all()
      pq.write_table(tbl, 'dict_table.parquet') # SEGFAULTS
      

      When writing record batch streams, if there are no nulls in an array, Arrow will put a placeholder nullptr instead of putting the full bitmap of 1s, when deserializing that stream, the bitmap for the nulls isn't populated and is left to being a nullptr. When attempting to write this table via pyarrow.parquet, you end up here in the parquet writer code which attempts to Cast the dictionary to a non-dictionary representation. Since the null count isn't checked before creating a BitmapReader, the BitmapReader is constructed with a nullptr for the bitmap_data, but a non-zero length which then segfaults in the constructor here because bitmap is null.

      So a simple check of the null count before constructing the BitmapReader avoids the segfault.

      Already filed PR 1896

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                zeroshade Matt Topol
                Reporter:
                zeroshade Matt Topol
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m