[SPARK-11284] ALS produces predictions as floats and should be double

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.5.1
• Component/s: ML
• Labels: ml recommender

Environment

All

• Assignee: Unassigned
• Reporter: Dominik Dahlem
• Created: 23/Oct/15 13:52
• Updated: 13/Mar/19 13:00
• Resolved: 27/Oct/15 04:26

Description

Using pyspark.ml and DataFrames, The ALS recommender cannot be evaluated using the RegressionEvaluator, because of a type mis-match between the model transformation and the evaluation APIs. One can work around this by casting the prediction column into double before passing it into the evaluator. However, this does not work with pipelines and cross validation.

Code and traceback below:

        als = ALS(rank=10, maxIter=30, regParam=0.1, userCol='userID', itemCol='movieID', ratingCol='rating')
        model = als.fit(training)
        predictions = model.transform(validation)
        evaluator = RegressionEvaluator(predictionCol='prediction', labelCol='rating')
        validationRmse = evaluator.evaluate(predictions, {evaluator.metricName: 'rmse'})

Traceback:
validationRmse = evaluator.evaluate(predictions,

{evaluator.metricName: 'rmse'}

)
File "/Users/dominikdahlem/software/spark-1.6.0-SNAPSHOT-bin-custom-spark/python/lib/pyspark.zip/pyspark/ml/evaluation.py", line 63, in evaluate
File "/Users/dominikdahlem/software/spark-1.6.0-SNAPSHOT-bin-custom-spark/python/lib/pyspark.zip/pyspark/ml/evaluation.py", line 94, in _evaluate
File "/Users/dominikdahlem/software/spark-1.6.0-SNAPSHOT-bin-custom-spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in _call_
File "/Users/dominikdahlem/projects/repositories/spark/python/pyspark/sql/utils.py", line 42, in deco
raise IllegalArgumentException(s.split(': ', 1)[1])
pyspark.sql.utils.IllegalArgumentException: requirement failed: Column prediction must be of type DoubleType but was actually FloatType.

Issue Links

• links to: [Github] Pull Request #9252 (dahlem)