Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4531

Cache serialized java objects instead of serialized python objects in MLlib

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0
    • Component/s: MLlib, PySpark
    • Labels:
      None
    • Target Version/s:

      Description

      The Pyrolite is pretty slow (comparing to the adhoc serializer in 1.1), it cause much performance regression in 1.2, because we cache the serialized Python object in JVM, deserialize them into Java object in each step.

      We should change to cache the deserialized JavaRDD instead of PythonRDD to avoid the deserialization of Pyrolite.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                davies Davies Liu
                Reporter:
                davies Davies Liu
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: