[HIVE-8359] Map containing null values are not correctly written in Parquet files

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.13.1
• Fix Version/s: 1.1.0
• Component/s: File Formats
• Assignee: Sergio Peña
• Reporter: Frédéric TERRAZZONI
• Created: 06/Oct/14 17:17
• Updated: 12/Feb/15 23:40
• Resolved: 19/Nov/14 19:06

Description

Tried write a map<string,string> column in a Parquet file. The table should contain :

{"key3":"val3","key4":null}
{"key3":"val3","key4":null}
{"key1":null,"key2":"val2"}
{"key3":"val3","key4":null}
{"key3":"val3","key4":null}

... and when you do a query like

SELECT * from mytable

We can see that the table is corrupted :

{"key3":"val3"}
{"key4":"val3"}
{"key3":"val2"}
{"key4":"val3"}
{"key1":"val3"}

I've not been able to read the Parquet file in our software afterwards, and consequently I suspect it to be corrupted.

For those who are interested, I generated this Parquet table from an Avro file.

Issue Links

• incorporates: HIVE-6914 parquet-hive cannot write nested map (map value is map)
• is blocked by: HIVE-8577 Cannot deserialize Avro schema with a map<string,string> with null values
• is duplicated by: HIVE-6994 parquet-hive createArray strips null elements
• links to: Review for HIVE-8359