[SPARK-6883] Fork pyspark's cloudpickle as a separate dependency

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: PySpark
• Labels: bulk-closed fork
• Assignee: Unassigned
• Reporter: Kyle Kelley
• Created: 13/Apr/15 17:21
• Updated: 21/May/19 04:15
• Resolved: 21/May/19 04:15

Description

IPython, pyspark, picloud/multyvac/cloudpipe all rely on cloudpickle from various sources (cloud, pyspark, and multyvac correspondingly). It would be great to have this as a separately maintained project that can:

• Work with Python3
• Add tests!
• Use higher order pickling (when on Python3)
• Be installed with pip

We're starting this off at the PyCon sprints under https://github.com/cloudpipe/cloudpickle. We'd like to coordinate with PySpark to make it work across all the above mentioned projects.