[HIVE-19016] Vectorization and Parquet: Disable vectorization for nested complex types

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.1.0, 4.0.0-alpha-1
• Component/s: Hive
• Assignee: Matt McCline
• Reporter: Matt McCline
• Created: 21/Mar/18 23:45
• Updated: 21/Oct/22 06:59
• Resolved: 22/Jun/18 15:53

Description

Original title: Vectorization and Parquet: When vectorized, parquet_nested_complex.q produces RuntimeException: Unsupported type used

 

Adding "SET hive.vectorized.execution.enabled=true;" to parquet_nested_complex.q triggers this call stack:

Caused by: java.lang.RuntimeException: Unsupported type used in list:array<array<array<array<array<array<array<array<array<array<array<array<array<array<array<array<array<array<array<array<array<array<int>>>>>>>>>>>>>>>>>>>>>>
	at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkListColumnSupport(VectorizedParquetRecordReader.java:589) ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:525) ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440) ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401) ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353) ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92) ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360) ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]

FYI: vihangk1

Issue Links

• is related to: HIVE-19015 Vectorization and Parquet: When vectorized, parquet_map_of_arrays_of_ints.q gets a ClassCastException
• relates to: HIVE-18576 Support to read nested complex type with Parquet in vectorization mode
• links to: RB #67654