Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
Description
It appears that the RowGroupMetaData file_offset property is being set to the same value as the first ColumnMetaData file_offset property in https://github.com/apache/arrow/blob/8e43f23dcc6a9e630516228f110c48b64d13cec6/cpp/src/parquet/metadata.cc#L1557-L1565
This is not consistent with the definition of these properties given in the Thrift file: https://github.com/apache/arrow/blob/master/cpp/src/parquet/parquet.thrift
struct ColumnChunk { ... /** Byte offset in file_path to the ColumnMetaData **/ 2: required i64 file_offset ... } ... struct RowGroup { ... /** Byte offset from beginning of file to first page (data or dictionary) * in this row group **/ 5: optional i64 file_offset ... }
This is causing issues when trying to read the file with the parquet-mr libraries, because the RowGroup's file offset is used to determine whether a RowGroup exists within a given file split: https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java#L1226-L1251
This issue is therefore resulting in Parquet files which cannot be read as the metadata is incorrect.
Attachments
Issue Links
- links to