[SPARK-33203] Pyspark ml tests failing with rounding errors

• Type: Test
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 3.1.0
• Component/s: ML, PySpark
• Assignee: Alessandro Patti
• Reporter: Alessandro Patti
• Created: 21/Oct/20 07:45
• Updated: 22/Oct/20 01:15
• Resolved: 22/Oct/20 01:14

Description

The tests pyspark.ml.recommendation and pyspark.ml.tests.test_algorithms occasionally fail (depends on environment) with

File "/home/jenkins/python/pyspark/ml/tests/test_algorithms.py", line 96, in test_raw_and_probability_prediction
 self.assertTrue(np.allclose(result.rawPrediction, expected_rawPrediction, atol=1))
AssertionError: False is not true

File "/home/jenkins/python/pyspark/ml/recommendation.py", line 256, in _main_.ALS
Failed example:
 predictions[0]
Expected:
 Row(user=0, item=2, newPrediction=0.6929101347923279)
Got:
 Row(user=0, item=2, newPrediction=0.6929104924201965)
...

Issue Links

• links to: [Github] Pull Request #30104 (AlessandroPatti)
• links to: [Github] Pull Request #30104 (AlessandroPatti)