Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32094

Patch cloudpickle.py with typing module side-effect fix

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersStop watchingWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.6, 3.0.0
    • 3.1.0
    • PySpark
    • None

    Description

      Pyspark's cloudpickle.py and versions of cloudpickle below 1.3.0 interfere with dill unpickling because they define types.ClassType, which is undefined in dill. This results in the following error:

      Traceback (most recent call last):
        File "/usr/local/lib/python3.6/site-packages/apache_beam/internal/pickler.py", line 279, in loads
          return dill.loads(s)
        File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 317, in loads
          return load(file, ignore)
        File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 305, in load
          obj = pik.load()
        File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 577, in _load_type
          return _reverse_typemap[name]
      KeyError: 'ClassType'

      (See https://github.com/cloudpipe/cloudpickle/issues/82)

      This was fixed for cloudpickle 1.3.0+ (https://github.com/cloudpipe/cloudpickle/pull/337), but PySpark's cloudpickle.py doesn't have this change yet.

       

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            hyukjin.kwon Hyukjin Kwon Assign to me
            codesue Suzen Fylke
            Votes:
            1 Vote for this issue
            Watchers:
            4 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment