[SPARK-3541] Improve ALS internal storage

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Component/s: MLlib
• Target Version/s: 1.3.0
• Assignee: Xiangrui Meng
• Reporter: Xiangrui Meng
• Created: 16/Sep/14 02:37
• Updated: 23/Jan/15 06:09
• Resolved: 23/Jan/15 06:09

Description

The internal storage of ALS uses many small objects, which increases the GC pressure and makes ALS difficult to scale to very large scale, e.g., 50 billion ratings. In such cases, the full GC may take more than 10 minutes to finish. That is longer than the default heartbeat timeout and hence executors will be removed under default settings.

We can use primitive arrays to reduce the number of objects significantly. This requires big change to the ALS implementation.

Issue Links

• blocks: SPARK-3735 Sending the factor directly or AtA based on the cost in ALS
• is blocked by: SPARK-4084 Reuse sort key in Sorter
• links to: [Github] Pull Request #3720 (mengxr)