Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-6678

[C++] Regression in Parquet file compatibility introduced by ARROW-3246

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.15.0
    • Component/s: C++

      Description

      I randomly discovered that this script fails after applying the patch for ARROW-3246

      https://github.com/apache/arrow/commit/2ba0566b29312e84fafc987fd8dc9664748be96a

      import pandas as pd
      import pyarrow as pa
      import pyarrow.parquet as pq
      import fastparquet as fp
      
      df = pd.util.testing.makeDataFrame()
      
      pq.write_table(pa.table(df), 'test.parquet')
      
      fp.ParquetFile('test.parquet')
      

      with

      Traceback (most recent call last):
        File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/api.py", line 110, in __init__
          with open_with(fn2, 'rb') as f:
        File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/util.py", line 38, in default_open
          return open(f, mode)
      NotADirectoryError: [Errno 20] Not a directory: 'test.parquet/_metadata'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "test.py", line 10, in <module>
          fp.ParquetFile('test.parquet')
        File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/api.py", line 116, in __init__
          self._parse_header(f, verify)
        File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/api.py", line 135, in _parse_header
          fmd = read_thrift(f, parquet_thrift.FileMetaData)
        File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/thrift_structures.py", line 25, in read_thrift
          obj.read(pin)
        File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/parquet_thrift/parquet/ttypes.py", line 1929, in read
          iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec])
      UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 0: invalid start byte
      

      I don't recall making any metadata-related changes but I'm going to review the patch to see if I can narrow down where the problem is to see whether it's a bug with Arrow/parquet-cpp or with the third party library

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                wesm Wes McKinney
                Reporter:
                wesm Wes McKinney
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 40m
                  2h 40m