[SPARK-43297] Make improvement to LocalKMeans

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: MLlib
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: wenweijian
• Created: 26/Apr/23 17:00
• Updated: 22/Sep/23 00:17

Description

There are two initializationMode in Kmeans, random mode and parallel mode.

The ParallelMode is using kmeansPlusPlus to generate the centers point, but the kMeansPlusPlus is a local method which runs in the driver.

If the scale of points is huge, the kMeansPlusPlus will run for a long time, because it is a single thread method running in the driiver.

We can make this method run in parallel to make it faster, such as using Parallel collections. 

 

 

 

Issue Links

• links to: [Github] Pull Request #41384 (wenwj0)
• links to: GitHub Pull Request #41384