Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-402

Apache Pig cannot store Map data type into Parquet format

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 1.6.0, 1.8.1
    • Fix Version/s: None
    • Component/s: parquet-pig
    • Labels:
      None

      Description

      Trying to store simple map with two entries gives me following exception:

      table_with_map_data: {my_map: map[]}
      2015-12-10 11:58:54,478 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
      2015-12-10 11:58:54,498 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2999: Unexpected internal error. Invalid map Schema, schema should contain exactly one field: my_map: map

      For example taking any input and doing this gives me the exception:

      table_with_map_data = FOREACH random_data GENERATE TOMAP('123', 'hello', '456', 'world') as (my_map);
      DESCRIBE table_with_map_data;
      STORE table_with_map_data INTO '...' USING ParquetStorer();

      I'm using latest version of Pig: Apache Pig version 0.15.0 (r1682971) compiled Jun 01 2015, 11:44:35
      and Parquet: parquet-pig-bundle-1.6.0.jar

      EDIT: I noticed Parquet 1.8.1 is out. I switched to it and were forced to update the pig script to use full path with ParquetStorer. However this gives me same error as 1.6.0.

      STORE table_with_map_data INTO '/Users/jerry/tmp/parquet/output/parquet' USING org.apache.parquet.pig.ParquetStorer();

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jerry.ylilammi Jerry Ylilammi
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: