The method calculateNumberOfPartitions() uses Int, not Long (unlike the MLlib version), so it is very easily to have an overflow in calculating the number of partitions for ML persistence.
- is related to
-
SPARK-19247 Improve ml word2vec save/load scalability
-
- Resolved
-
- links to