[SPARK-32094] Patch cloudpickle.py with typing module side-effect fix

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.6, 3.0.0
• Fix Version/s: 3.1.0
• Component/s: PySpark
• Assignee: Hyukjin Kwon
• Reporter: Suzen Fylke
• Created: 24/Jun/20 19:19
• Updated: 12/Dec/22 18:10
• Resolved: 17/Jul/20 02:49

Description

Pyspark's cloudpickle.py and versions of cloudpickle below 1.3.0 interfere with dill unpickling because they define types.ClassType, which is undefined in dill. This results in the following error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/apache_beam/internal/pickler.py", line 279, in loads
    return dill.loads(s)
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 317, in loads
    return load(file, ignore)
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 305, in load
    obj = pik.load()
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 577, in _load_type
    return _reverse_typemap[name]
KeyError: 'ClassType'

(See https://github.com/cloudpipe/cloudpickle/issues/82)

This was fixed for cloudpickle 1.3.0+ (https://github.com/cloudpipe/cloudpickle/pull/337), but PySpark's cloudpickle.py doesn't have this change yet.

 

Issue Links

• links to: [Github] Pull Request #28950 (codesue)
• links to: [Github] Pull Request #28950 (codesue)
• links to: [Github] Pull Request #29114 (HyukjinKwon)