[SPARK-49101] Cannot grow BufferHolder by size 86352 because the size after growing exceeds size limitation 2147483632

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.1
• Component/s: PySpark

Environment

Python 3.10.13

Pyspark version 3.4.1

 

• Assignee: Unassigned
• Reporter: Sonal Prasad
• Created: 03/Aug/24 00:42
• Updated: 09/Aug/24 22:53

Description

java.lang.IllegalArgumentException:
Cannot grow BufferHolder by size 86352 because the size after growing exceeds
size limitation 2147483632