[IMPALA-5037] Change default Parquet array resolution according to Parquet standard.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.8.0, Impala 2.9.0
• Fix Version/s: Impala 3.0
• Component/s: Backend
• Labels: include-in-v3 incompatibility
• Target Version: Impala 3.0
• Assignee: Alexander Behm
• Reporter: Alexander Behm
• Created: 07/Mar/17 05:56
• Updated: 12/Feb/18 21:49
• Resolved: 06/Feb/18 23:11

Description

With IMPALA-4725 we've introduced query options to control the field resolution behavior when scanning Parquet files with nested arrays. The current default strategy currently tries to auto-detect the array encoding within Parquet files, but this strategy can sometimes subtly go wrong and return incorrect results due to the inherent ambiguity of the 2/3-level encoding schemes in Parquet.

We should switch the default resolution strategy according to the Parquet standard 3-level encoding, instead of the current auto-detect.

Issue Links

• breaks: IMPALA-6489 ASAN use-after-poison in impala::HdfsScanner::InitTupleFromTemplate