[ARROW-6678] [C++] Regression in Parquet file compatibility introduced by ARROW-3246

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.15.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/23026
• Assignee: Wes McKinney
• Reporter: Wes McKinney
• Created: 24/Sep/19 23:02
• Updated: 11/Jan/23 07:48
• Resolved: 25/Sep/19 04:37

Description

I randomly discovered that this script fails after applying the patch for ARROW-3246

https://github.com/apache/arrow/commit/2ba0566b29312e84fafc987fd8dc9664748be96a

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
import fastparquet as fp

df = pd.util.testing.makeDataFrame()

pq.write_table(pa.table(df), 'test.parquet')

fp.ParquetFile('test.parquet')

with

Traceback (most recent call last):
  File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/api.py", line 110, in __init__
    with open_with(fn2, 'rb') as f:
  File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/util.py", line 38, in default_open
    return open(f, mode)
NotADirectoryError: [Errno 20] Not a directory: 'test.parquet/_metadata'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 10, in <module>
    fp.ParquetFile('test.parquet')
  File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/api.py", line 116, in __init__
    self._parse_header(f, verify)
  File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/api.py", line 135, in _parse_header
    fmd = read_thrift(f, parquet_thrift.FileMetaData)
  File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/thrift_structures.py", line 25, in read_thrift
    obj.read(pin)
  File "/home/wesm/miniconda/envs/arrow-3.7/lib/python3.7/site-packages/fastparquet/parquet_thrift/parquet/ttypes.py", line 1929, in read
    iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec])
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 0: invalid start byte

I don't recall making any metadata-related changes but I'm going to review the patch to see if I can narrow down where the problem is to see whether it's a bug with Arrow/parquet-cpp or with the third party library

Issue Links

• links to: GitHub Pull Request #5493