Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-6914

parquet-hive cannot write nested map (map value is map)

    XMLWordPrintableJSON

Details

    Description

      // table schema (identical for both plain text version and parquet version)
      desc hive> desc text_mmap;
      m map>

      // sample nested map entry
      {"level1":{"level2_key1":"value1","level2_key2":"value2"}}

      The following query will fail,
      insert overwrite table parquet_mmap select * from text_mmap;

      Caused by: parquet.io.ParquetEncodingException: This should be an ArrayWritable or MapWritable: org.apache.hadoop.hive.ql.io.parquet.writable.BinaryWritable@f2f8106
      at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeData(DataWritableWriter.java:85)
      at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeArray(DataWritableWriter.java:118)
      at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeData(DataWritableWriter.java:80)
      at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeData(DataWritableWriter.java:82)
      at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:55)
      at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
      at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
      at parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:115)
      at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:81)
      at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:37)
      at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:77)
      at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:90)
      at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:622)
      at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
      at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
      at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
      at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
      at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
      at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:540)
      ... 9 more

      Attachments

        1. NestedMap.parquet
          0.6 kB
          Ryan Blue
        2. HIVE-6914.4.patch
          4 kB
          Sergio Peña
        3. HIVE-6914.3.patch
          4 kB
          Ryan Blue
        4. HIVE-6914.2.patch
          30 kB
          Mickael Lacour
        5. HIVE-6914.1.patch
          28 kB
          Mickael Lacour
        6. HIVE-6914.1.patch
          4 kB
          Ryan Blue

        Issue Links

          Activity

            People

              rdblue Ryan Blue
              tongjie Tongjie Chen
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: