[SPARK-21050] ml word2vec write has overflow issue in calculating numPartitions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.0, 2.3.0
• Component/s: ML
• Target Version/s: 2.2.1, 2.3.0
• Assignee: Joseph K. Bradley
• Reporter: Joseph K. Bradley
• Created: 11/Jun/17 00:34
• Updated: 03/Jul/17 11:15
• Resolved: 12/Jun/17 21:27

Description

The method calculateNumberOfPartitions() uses Int, not Long (unlike the MLlib version), so it is very easily to have an overflow in calculating the number of partitions for ML persistence.

Issue Links

• is related to: SPARK-19247 Improve ml word2vec save/load scalability
• links to: [Github] Pull Request #18265 (jkbradley)