[ARROW-14196] [C++][Parquet] Default to compliant nested types in Parquet writer

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: C++, Parquet
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/29781
• Assignee: Will Jones
• Reporter: Will Jones
• Created: 01/Oct/21 13:45
• Updated: 11/Jan/23 08:38

Description

In C++ there is already an option to get the "compliant_nested_types" (to have the list columns follow the Parquet specification), and ARROW-11497 exposed this option in Python.

This is still set to False by default, but in the source it says "TODO: At some point we should flip this.", and in ARROW-11497 there was also some discussion about what it would take to change the default.

cc emkornfield apitrou

Issue Links

• is related to: ARROW-11497 [Python] pyarrow parquet writer for list does not conform with Apache Parquet specification
• relates to: ARROW-14286 [Python][Parquet] Allow to select columns of a list field without requiring the list component names
• links to: GitHub Pull Request #14651