Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6883

Fork pyspark's cloudpickle as a separate dependency

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete CommentsDelete
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • None
    • None
    • PySpark

    Description

      IPython, pyspark, picloud/multyvac/cloudpipe all rely on cloudpickle from various sources (cloud, pyspark, and multyvac correspondingly). It would be great to have this as a separately maintained project that can:

      • Work with Python3
      • Add tests!
      • Use higher order pickling (when on Python3)
      • Be installed with pip

      We're starting this off at the PyCon sprints under https://github.com/cloudpipe/cloudpickle. We'd like to coordinate with PySpark to make it work across all the above mentioned projects.

      Attachments

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            rgbkrk Kyle Kelley
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment