[SPARK-5490] KMeans costs can be incorrect if tasks need to be rerun

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.3.0
• Component/s: MLlib
• Labels: bulk-closed clustering
• Assignee: Unassigned
• Reporter: Sandy Ryza
• Created: 29/Jan/15 23:29
• Updated: 21/May/19 05:36
• Resolved: 21/May/19 05:36

Description

KMeans uses accumulators to compute the cost of a clustering at each iteration.

Each time a ShuffleMapTask completes, it increments the accumulators at the driver. If a task runs twice because of failures, the accumulators get incremented twice.

KMeans uses accumulators in ShuffleMapTasks. This means that a task's cost can end up being double-counted.

Issue Links

• relates to: SPARK-732 Recomputation of RDDs may result in duplicated accumulator updates