Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13632

Hive failing on insert empty array into parquet table

    XMLWordPrintableJSON

Details

    Description

      The insert will fail with following stack:

      by: parquet.io.ParquetEncodingException: empty fields are illegal, the field should be ommited completely instead
      	at parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:271)
      	at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$ListDataWriter.write(DataWritableWriter.java:271)
      	at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$GroupDataWriter.write(DataWritableWriter.java:199)
      	at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$MessageDataWriter.write(DataWritableWriter.java:215)
      	at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:88)
      	at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
      	at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
      	at parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:116)
      	at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
      	at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
      	at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
      	at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
      	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:697)
      

      Reproduce:

      create table test_small (
      key string,
      arrayValues array<string>)
      stored as parquet;
      insert into table test_small select 'abcd', array() from src limit 1;
      

      Attachments

        1. HIVE-13632.1.patch
          8 kB
          Yongzhi Chen
        2. HIVE-13632.2.patch
          12 kB
          Yongzhi Chen
        3. HIVE-13632.3.patch
          14 kB
          Yongzhi Chen

        Issue Links

          Activity

            People

              ychena Yongzhi Chen
              ychena Yongzhi Chen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: