[ARROW-1644] [C++][Parquet] Read and write nested Parquet data with a mix of struct and list nesting levels

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 2.0.0
• Component/s: C++, Python
• Labels: parquet pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/17654
• Assignee: Micah Kornfield
• Reporter: DB Tsai
• Created: 05/Oct/17 00:43
• Updated: 11/Jan/23 07:16
• Resolved: 22/Oct/20 03:13

Description

We have many nested parquet files generated from Apache Spark for ranking problems, and we would like to load them in python for other programs to consume.

The schema looks like

root
 |-- profile_id: long (nullable = true)
 |-- country_iso_code: string (nullable = true)
 |-- items: array (nullable = false)
 |    |-- element: struct (containsNull = false)
 |    |    |-- show_title_id: integer (nullable = true)
 |    |    |-- duration: double (nullable = true)

And when I tried to load it with nightly build pyarrow on Oct 4, 2017, I got the following error.

Python 3.6.2 |Anaconda, Inc.| (default, Sep 30 2017, 18:42:57) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> import pandas as pd
>>> import pyarrow as pa
>>> import pyarrow.parquet as pq
>>> table2 = pq.read_table('part-00000')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dbt/miniconda3/lib/python3.6/site-packages/pyarrow/parquet.py", line 823, in read_table
    use_pandas_metadata=use_pandas_metadata)
  File "/home/dbt/miniconda3/lib/python3.6/site-packages/pyarrow/parquet.py", line 119, in read
    nthreads=nthreads)
  File "_parquet.pyx", line 466, in pyarrow._parquet.ParquetReader.read_all
  File "error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: lists with structs are not supported.

I somehow get the impression that after https://issues.apache.org/jira/browse/PARQUET-911 is merged, we should be able to load the nested parquet in pyarrow.

Any insight about this?

Thanks.

Issue Links

• duplicates: ARROW-6737 Nested column branch had multiple children
• duplicates: ARROW-7845 [C++] Reading list from parquet files
• is related to: ARROW-2587 [Python] Unable to write StructArrays with multiple children to parquet
• is related to: ARROW-5799 [Python] Fail to write nested data to Parquet via BigQuery API
• links to: GitHub Pull Request #462

Sub-Tasks

• [C++] Create performance benchmark for parquet reading: Closed
• [C++] Rebase https://github.com/apache/parquet-cpp/pull/462# onto arrow repo: Closed
• [C++][Parquet] Add a basic disabled unit test to excercise nesting functionality: Resolved
• [C++][Parquet] Incorporate new level generation logic in parquet write path with a flag to revert back to old logic: Resolved
• [C++] Add schema conversion support for map type: Resolved
• [C++][Parquet] Add a new level builder capable of handling nested data: Resolved
• [C++] Refactor DefLevelsToBitmap: Resolved
• [C++] Cleanup Parquet Arrow Schema code: Resolved
• [C++] Expose a ReadValuesSpaced method that accepts a validity bitmap.: Closed
• [C++] Create unified schema resolution code for Array reconstruction.: Resolved
• [C++] Add hand-crafted Parquet to Arrow reconstruction test for nested reading: Resolved
• [C++][Parquet] Generalize existing null bitmap generation: Open
• [C++] Implement basic array-by-array reassembly logic: Resolved
• [C++][Parquet] Create randomized nested data generation round trip read/write unit tests: Open
• [C++][Parquet] Add support for schema translation from parquet nodes back to arrow for missing types: Resolved
• [C++][Parquet] Implement non-vectorized array reconstruction logic.: Open
• [C++][Parquet] Add EngineVersion to properties to allow for toggling new vs old logic: Closed
• [Python][Parquet] Expose EngineVersion in python arrow reader properties: Closed
• [C++][Parquet] Create nested reading benchmarks: Resolved
• [C++] Add Parquet-Arrow roundtrip tests for nested data: Resolved
• [C++] Investigate performance of LevelsToBitmap without BMI2: Resolved
• [C++][Parquet] Create reading benchmarks for 2-level nested data: Resolved