[IMPALA-2272] Parquet scanner always materializes NULL for empty collections

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 2.3.0
• Fix Version/s: Impala 4.1.0
• Component/s: Backend
• Labels: complextype nested_types
• Epic Link: Fix complex type related issues
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Skye Wanderman-Milne
• Created: 31/Aug/15 22:36
• Updated: 17/Feb/22 19:48
• Resolved: 17/Feb/22 19:48

Description

Currently the Parquet scanner will always materialize a NULL slot for an empty collection, rather than an empty ArrayValue/CollectionValue. It is not currently possible to write a query that exposes this bug (i.e. it's not possible to write a query that distinguishes between an empty and NULL collection), but it will be once we add expressions that take collections as input (e.g. "select array_column is null from tbl").

We have this bug because the parquet scanner only looks at the repeated field of an array, not the containing group field. To fix it, it will have to consider the def/rep levels of both.

Issue Links

• relates to: IMPALA-9498 Allow array type in SELECT list for Parquet tables