Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21542

Helper functions for custom Python Persistence

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.2.0
    • 2.3.0
    • ML, PySpark
    • None

    Description

      Currently, there is no way to easily persist Json-serializable parameters in Python only. All parameters in Python are persisted by converting them to Java objects and using the Java persistence implementation. In order to facilitate the creation of custom Python-only pipeline stages, it would be good to have a Python-only persistence framework so that these stages do not need to be implemented in Scala for persistence.

      This task involves:

      • Adding implementations for DefaultParamsReadable, DefaultParamsWriteable, DefaultParamsReader, and DefaultParamsWriter in pyspark.

      Attachments

        Issue Links

          relates to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-17025 Cannot persist PySpark ML Pipeline model that includes custom Transformer

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Web Link PR 18742

          Activity

            People

              ajaysaini Ajay Saini
              ajaysaini Ajay Saini
              Joseph K. Bradley Joseph K. Bradley
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: