[PARQUET-62] DictionaryValuesWriter dictionaries are corrupted by user changes.

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: parquet-mr
• Assignee: Ryan Blue
• Reporter: Ryan Blue
• Created: 02/Aug/14 03:20
• Updated: 07/Jul/15 15:44
• Resolved: 20/Aug/14 21:02

Description

DictionaryValuesWriter passes incoming Binary objects directly to Object2IntMap to accumulate dictionary values. If the arrays backing the Binary objects passed in are reused by the caller, then the values are corrupted but still written without an error.

Because Hadoop reuses objects passed to mappers and reducers, this can happen easily. For example, Avro reuses the byte arrays backing Utf8 objects, which parquet-avro passes wrapped in a Binary object to writeBytes.

The fix is to make defensive copies of the values passed to the Dictionary writer code. I think this only affects the Binary dictionary classes because Strings, floats, longs, etc. are immutable.

Issue Links

• relates to: PARQUET-326 Binary statistics are invalid if buffers are reused