[SPARK-43949] Upgrade Cloudpickle to 2.2.1

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.2, 3.4.0, 3.5.0
• Fix Version/s: 3.4.1, 3.5.0
• Component/s: PySpark
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 02/Jun/23 08:16
• Updated: 02/Jun/23 10:40
• Resolved: 02/Jun/23 10:40

Description

Cloudpickle 2.2.1 has a fix for named tuple issue (https://github.com/cloudpipe/cloudpickle/issues/460). PySpark relies on namedtuple heavily especially for RDD. We should upgrade and fix it.