Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13625

PySpark-ML method to get list of params for an obj should not check property attr

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: ML, PySpark
    • Labels:
      None
    • Target Version/s:

      Description

      In PySpark params._init_.py, the method Param.params() returns a list of Params belonging to that object. This method should not check an attribute to be an instance of Param if it is a property (uses the @property decorator). This causes the property to be invoked to 'get' the attribute, and that can lead to an error, depending on the property. If an attribute is a property it will not be an ML Param, so no need to check it.

      I came across this in working on SPARK-13430 while adding LinearRegressionModel.summary as a property to give a training summary, similar to the Scala API. It is possible that a training summary does not exist and will then raise an exception if the summary property is invoked.

      Calling getattr(self, x) will cause the property to be invoked if x is a property. To fix this, just need to check if it a class property before making the call to getattr() in Param.params().

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bryanc Bryan Cutler
                Reporter:
                bryanc Bryan Cutler
                Shepherd:
                Joseph K. Bradley
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: