Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4898

Replace cloudpickle with Dill

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • None
    • None
    • PySpark
    • None

    Description

      We should consider replacing our modified version of cloudpickle with Dill, since it supports both Python 2 and 3 and might do a better job of handling certain corner-cases.

      I attempted to do this a few months ago but ran into cases where Dill had issues pickling objects defined in doctests, which broke our test suite: https://github.com/uqfoundation/dill/issues/50. This issue may have been resolved now; I haven't checked.

      Attachments

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-4897 Python 3 support

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Activity

            People

              Unassigned Unassigned
              joshrosen Josh Rosen
              Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: