[HADOOP-11901] BytesWritable fails to support 2G chunks due to integer overflow

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Hadoop Flags: Reviewed
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 01/May/15 22:48
• Updated: 08/Jun/18 19:32
• Resolved: 19/Nov/15 21:00

Description

BytesWritable.setSize increases the buffer size by 1.5 each time ( * 3 / 2). This is an unsafe operation since it restricts the max size to ~700MB, since 700MB * 3 > 2GB.

I didn't write a test case for this case because in order to trigger this, I'd need to allocate around 700MB, which is pretty expensive to do in a unit test. Note that I didn't throw any exception in the case integer overflow as I didn't want to change that behavior (callers to this might expect a java.lang.NegativeArraySizeException).

Issue Links

• causes: HIVE-11105 NegativeArraySizeException from org.apache.hadoop.io.BytesWritable.setCapacity during serialization phase
• duplicates: MAPREDUCE-21 NegativeArraySizeException in reducer with new api