[IMPALA-4826] Impala should ignore the root schema's repetition in Parquet

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0
• Fix Version/s: Impala 2.11.0
• Component/s: Backend
• Labels: parquet ramp-up
• Target Version: Product Backlog
• Assignee: Gabor Kaszab
• Reporter: Tim Armstrong
• Created: 26/Jan/17 01:54
• Updated: 07/Sep/17 12:21
• Resolved: 07/Sep/17 12:21

Description

See https://issues.apache.org/jira/browse/PARQUET-843 . parquet-cpp was generating files that set the root schema's repetition to REPEATED, which threw off Impala's schema resolution so it couldn't read the file. PARQUET-843 includes an example file

The field description in parquet.thrift explicitly says that the root schema's repetition should be unset (https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L231) but it seems like other tools may write out various things there.

We should just ignore the repetition on the root schema, since it's meaningless.