-
Type:
Improvement
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.8.0, 3.0.0-alpha1
-
Component/s: None
-
Labels:None
-
Hadoop Flags:Reviewed
BytesWritable.setSize increases the buffer size by 1.5 each time ( * 3 / 2). This is an unsafe operation since it restricts the max size to ~700MB, since 700MB * 3 > 2GB.
I didn't write a test case for this case because in order to trigger this, I'd need to allocate around 700MB, which is pretty expensive to do in a unit test. Note that I didn't throw any exception in the case integer overflow as I didn't want to change that behavior (callers to this might expect a java.lang.NegativeArraySizeException).
- causes
-
HIVE-11105 NegativeArraySizeException from org.apache.hadoop.io.BytesWritable.setCapacity during serialization phase
-
- Resolved
-
- duplicates
-
MAPREDUCE-21 NegativeArraySizeException in reducer with new api
-
- Resolved
-