Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3491

Use pickle to serialize the data in MLlib Python

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.2.0
    • MLlib, PySpark
    • None

    Description

      Currently, we write the code for serialization/deserialization in Python and Scala manually, it can not scale to the big number of MLlib API.

      If the serialization could be done in pickle (using Pyrolite in JVM) in extensional way, then it should be much easy to add Python API for MLlib.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-4023 PySpark's stat.Statistics is broken

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SPARK-3486 Add PySpark support for Word2Vec

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #2378 [Github] Pull Request #2378 (davies)

          Activity

            People

              davies Davies Liu
              davies Davies Liu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: