Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-19016

Vectorization and Parquet: Disable vectorization for nested complex types

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.1.0, 4.0.0
    • Component/s: Hive
    • Labels:
      None

      Description

      Original title: Vectorization and Parquet: When vectorized, parquet_nested_complex.q produces RuntimeException: Unsupported type used

       

      Adding "SET hive.vectorized.execution.enabled=true;" to parquet_nested_complex.q triggers this call stack:

      Caused by: java.lang.RuntimeException: Unsupported type used in list:array<array<array<array<array<array<array<array<array<array<array<array<array<array<array<array<array<array<array<array<array<array<int>>>>>>>>>>>>>>>>>>>>>>
      	at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkListColumnSupport(VectorizedParquetRecordReader.java:589) ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
      	at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:525) ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
      	at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440) ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
      	at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401) ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
      	at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353) ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
      	at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92) ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
      	at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360) ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
      

      FYI: Vihang Karajgaonkar

        Attachments

        1. HIVE-19016.02.patch
          31 kB
          Matt McCline
        2. HIVE-19016.01.patch
          33 kB
          Matt McCline

          Issue Links

            Activity

              People

              • Assignee:
                mmccline Matt McCline
                Reporter:
                mmccline Matt McCline
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: