[SPARK-36464] Fix Underlying Size Variable Initialization in ChunkedByteBufferOutputStream for Writing Over 2GB Data

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.8, 3.0.3, 3.1.2, 3.3.0
• Fix Version/s: 3.2.0, 3.1.3, 3.0.4, 3.2.1
• Component/s: Spark Core
• Assignee: Kazuyuki Tanimura
• Reporter: Kazuyuki Tanimura
• Created: 10/Aug/21 05:20
• Updated: 10/Jan/22 23:35
• Resolved: 10/Aug/21 17:34

Description

The `size` method of `ChunkedByteBufferOutputStream` returns a `Long` value; however, the underlying `_size` variable is initialized as `Int`.

That causes an overflow and returns a negative size when over 2GB data is written into `ChunkedByteBufferOutputStream`

Issue Links

• links to: [Github] Pull Request #33690 (kazuyukitanimura)
• links to: [Github] Pull Request #33690 (kazuyukitanimura)
• links to: [Github] Pull Request #33697 (dongjoon-hyun)
• links to: [Github] Pull Request #34284 (LuciferYang)
• links to: [Github] Pull Request #34284 (LuciferYang)
• links to: [Github] Pull Request #34284 (LuciferYang)