[SPARK-12450] Un-persist broadcasted variables in KMeans

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.4.1, 1.5.2
• Fix Version/s: 1.5.3, 1.6.1, 2.0.0
• Component/s: MLlib
• Target Version/s: 1.5.3, 1.6.1, 2.0.0
• Assignee: R J Nowling
• Reporter: R J Nowling
• Created: 21/Dec/15 15:35
• Updated: 05/Jan/16 23:05
• Resolved: 05/Jan/16 23:05

Description

The broadcasted centers in KMeans are never un-persisted. As a result, memory usage accumulates with usage causing a memory leak.

Issue Links

• relates to: SPARK-10329 Cost RDD in k-means|| initialization is not storage-efficient
• relates to: SPARK-10354 First cost RDD shouldn't be cached in k-means|| and the following cost RDD should use MEMORY_AND_DISK
• relates to: SPARK-11560 Optimize KMeans implementation / remove 'runs' from implementation
• links to: [Github] Pull Request #10415 (rnowling)