[PARQUET-215] Parquet Thrift should discard records with unrecognized union members

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: parquet-mr
• Assignee: Unassigned
• Reporter: Alex Levenson
• Created: 11/Mar/15 07:28
• Updated: 07/Apr/15 20:46
• Resolved: 12/Mar/15 21:25

Description

When writing parquet-thrift files, when a thrift record with an unknown union member is encountered, it should be considered a bad record and discarded. Currently, because unions are treated as structs with one optional field per union member, parquet-thrift happily writes the empty struct, but then crashes in the read path when trying to read this record.

We should discard these records in the write path, just as we discard other unparseable records.