[HIVE-14826] Support vectorization for Parquet

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Ferdinand Xu
• Reporter: Ferdinand Xu
• Created: 23/Sep/16 06:53
• Updated: 08/Jan/18 23:01

Description

Parquet vectorized reader can improve both throughput and also leverages existing Hive vectorization execution engine. This is an umbrella ticket to track this feature.

Issue Links

• is related to: HIVE-18323 Vectorization: add the support of timestamp in VectorizedPrimitiveColumnReader for parquet
• relates to: HIVE-15468 Enhance the vectorized execution engine to support complex types

Sub-Tasks

• Implement Parquet vectorization reader for Primitive types: Resolved
• Micro benchmark for Parquet vectorized reader: Resolved
• Test the predicate pushing down support for Parquet vectorization read path: Patch Available
• Implement Parquet vectorization reader for Struct type: Resolved
• Support Nested Column Field Pruning for Parquet Vectorized Reader: Open
• Fix the NullPointer problem caused by split phase: Resolved
• Parquet vectorization doesn't work for tables with partition info: Closed
• ParquetFileReader should be closed to avoid resource leak: Closed
• Measure Performance for Parquet Vectorization Reader: Open
• When we enable Parquet Writer Version V2, hive throws an exception: Unsupported encoding: DELTA_BYTE_ARRAY.: Closed
• Add more q-tests for Hive-on-Spark with Parquet vectorized reader: Closed
• Add a config to turn off parquet vectorization: Closed
• Vectorized reader does not seem to be pushing down projection columns in certain code paths: Closed
• Remove Parquet specific code from VectorizedColumnReader: Open
• Parquet vectorization fails on tables with complex columns when there are no projected columns: Closed
• Vectorized reader does push down projection columns for index access schema: Resolved
• Implement Parquet vectorization reader for Array type: Closed
• Support column projection for index access when using Parquet Vectorization: Closed
• NPE during initialization of VectorizedParquetRecordReader when input split is null: Closed
• Implement Parquet vectorization reader for Map type: Closed
• Fix API call in VectorizedListColumnReader to get value from BytesColumnVector: Closed
• Support to read multiple level definition for Map type in Parquet file: Closed
• Support vectorization for INTERVAL_DAY_TIME type: Open
• Vectorization: add the support of timestamp in VectorizedPrimitiveColumnReader for parquet: Closed
• Fix ArrayIndexOutOfBoundsException for VectorizedListColumnReader: Closed
• Support schema evolution in Parquet Vectorization reader: Closed
• Support to read nested complex type with Parquet in vectorization mode: Open