[HIVE-15055] Column pruning for nested fields in Parquet

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: Logical Optimizer, Physical Optimizer, Serializers/Deserializers
• Labels: performance
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 25/Oct/16 17:03
• Updated: 23/Dec/16 19:47

Description

Some columnar file formats such as Parquet store fields in struct type also column by column using encoding described in Google Dramel pager. It's very common in big data where data are stored in structs while queries only needs a subset of the the fields in the structs. However, presently Hive still needs to read the whole struct regardless whether all fields are selected. Therefore, pruning unwanted sub-fields in struct or nested fields at file reading time would be a big performance boost for such scenarios.

Sub-Tasks

• Support column pruning for struct fields in select statement: Resolved
• Support index shifting for struct fields: Resolved
• Nested column pruning: add pruned column paths to explain output: Resolved
• Nested column pruning: support all operators: Resolved
• Fixing shifting index issue related to object inspector caching: Resolved
• Nested column pruning: be more aggressive on RS value expressions: Open
• Nested column pruning: don't prune paths when a SerDe is used only for serializing: Resolved
• Nested column pruning: fix issue when selecting struct field from array/map element: Resolved
• Nested column pruning: fix issue when selecting struct field from array/map element (part 2): Resolved
• Nested column pruning: further prune the case for SELECT array.struct_field: Open
• Nested column pruning: more optimization on lateral view: Open