[SPARK-21799] KMeans performance regression (5-6x slowdown) in Spark 2.2

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.2.0
• Component/s: MLlib
• Assignee: Unassigned
• Reporter: Siddharth Murching
• Created: 21/Aug/17 17:57
• Updated: 11/Sep/17 21:05
• Resolved: 11/Sep/17 21:05

Description

I've been running KMeans performance tests using spark-sql-perf and have noticed a regression (slowdowns of 5-6x) when running tests on large datasets in Spark 2.2 vs 2.1.

The test params are:

• Cluster: 510 GB RAM, 16 workers
• Data: 1000000 examples, 10000 features

After talking to josephkb, the issue seems related to the changes in SPARK-18356 introduced in this PR.

It seems `df.cache()` doesn't set the storageLevel of `df.rdd`, so `handlePersistence` is true even when KMeans is run on a cached DataFrame. This unnecessarily causes another copy of the input dataset to be persisted.

As of Spark 2.1 (JIRA link) `df.storageLevel` returns the correct result after calling `df.cache()`, so I'd suggest replacing instances of `df.rdd.getStorageLevel` with df.storageLevel` in MLlib algorithms (the same pattern shows up in LogisticRegression, LinearRegression, and others). I've verified this behavior in this notebook

Issue Links

• duplicates: SPARK-18608 Spark ML algorithms that check RDD cache level for internal caching double-cache data
• links to: [Github] Pull Request #19107 (WeichenXu123)