Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21542

Helper functions for custom Python Persistence

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: ML, PySpark
    • Labels:
      None

      Description

      Currently, there is no way to easily persist Json-serializable parameters in Python only. All parameters in Python are persisted by converting them to Java objects and using the Java persistence implementation. In order to facilitate the creation of custom Python-only pipeline stages, it would be good to have a Python-only persistence framework so that these stages do not need to be implemented in Scala for persistence.

      This task involves:

      • Adding implementations for DefaultParamsReadable, DefaultParamsWriteable, DefaultParamsReader, and DefaultParamsWriter in pyspark.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ajaysaini Ajay Saini
                Reporter:
                ajaysaini Ajay Saini
                Shepherd:
                Joseph K. Bradley
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: