[SPARK-15447] Performance test for ALS in Spark 2.0

• Type: Task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: ML
• Labels: QA
• Target Version/s: 2.0.0
• Assignee: Nicholas Pentreath
• Reporter: Xiangrui Meng
• Created: 20/May/16 16:48
• Updated: 17/Jun/16 12:34
• Resolved: 17/Jun/16 12:34

Description

We made several changes to ALS in 2.0. It is necessary to run some tests to avoid performance regression. We should test (synthetic) datasets from 1 million ratings to 1 billion ratings.

cc mlnick holdenk Do you have time to run some large-scale performance tests?

Links:
Results spreadsheet
Raw results for SPARK-14891
Raw results for SPARK-6716

Issue Links

• relates to: SPARK-14891 ALS in ML never validates input schema
• relates to: SPARK-6717 Clear shuffle files after checkpointing in ALS