[PARQUET-402] Apache Pig cannot store Map data type into Parquet format

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.6.0, 1.8.1
• Component/s: parquet-pig
• Assignee: Unassigned
• Reporter: Jerry Ylilammi
• Created: 10/Dec/15 10:53
• Updated: 12/Dec/15 23:57
• Resolved: 10/Dec/15 14:22

Description

Trying to store simple map with two entries gives me following exception:

table_with_map_data: {my_map: map[]}
2015-12-10 11:58:54,478 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-12-10 11:58:54,498 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2999: Unexpected internal error. Invalid map Schema, schema should contain exactly one field: my_map: map

For example taking any input and doing this gives me the exception:

table_with_map_data = FOREACH random_data GENERATE TOMAP('123', 'hello', '456', 'world') as (my_map);
DESCRIBE table_with_map_data;
STORE table_with_map_data INTO '...' USING ParquetStorer();

I'm using latest version of Pig: Apache Pig version 0.15.0 (r1682971) compiled Jun 01 2015, 11:44:35
and Parquet: parquet-pig-bundle-1.6.0.jar

EDIT: I noticed Parquet 1.8.1 is out. I switched to it and were forced to update the pig script to use full path with ParquetStorer. However this gives me same error as 1.6.0.

STORE table_with_map_data INTO '/Users/jerry/tmp/parquet/output/parquet' USING org.apache.parquet.pig.ParquetStorer();