[SPARK-38243] Unintended exception thrown in pyspark.ml.LogisticRegression.getThreshold

• Type: Bug
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 2.4.0, 3.1.0, 3.2.0, 3.3.0
• Fix Version/s: 3.3.0
• Component/s: ML, PySpark
• Assignee: Maciej Szymkiewicz
• Reporter: Maciej Szymkiewicz
• Created: 17/Feb/22 15:48
• Updated: 18/Feb/22 17:09
• Resolved: 18/Feb/22 17:08

Description

If LogisticRegression.getThreshold is called with model having multiple thresholds we suppose to raise an exception,

ValueError: Logistic Regression getThreshold only applies to binary classification ...

However, thresholds (List[float]) are incorrectly passed to str.join, resulting in unintended TypeError

>>> from pyspark.ml.classification import LogisticRegression
... 
... model = LogisticRegression(thresholds=[1.0, 2.0, 3.0])
>>> model.getThreshold()
Traceback (most recent call last):
  Input In [7] in <module>
    model.getThreshold()
  File /path/to/spark/python/pyspark/ml/classification.py:1003 in getThreshold
    + ",".join(ts)
Type Error: sequence item 0: expected str instance, float found

Issue Links

• links to: [Github] Pull Request #35558 (zero323)
• links to: [Github] Pull Request #35558 (zero323)