[PARQUET-2052] Integer overflow when writing huge binary using dictionary encoding

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 21/May/21 01:57
• Updated: 06/Jul/22 22:27
• Resolved: 26/May/21 07:43

Description

To check whether it should fallback to plain encoding, DictionaryValuesWriter currently use two variables: dictionaryByteSize and maxDictionaryByteSize, both of which are integer. This will cause issue when one first writes a relatively small binary within the threshold and then write a huge string which cause dictionaryByteSize overflow and becoming negative.

Issue Links

• relates to: PARQUET-2164 CapacityByteArrayOutputStream overflow while writing causes negative row group sizes to be written