[SPARK-7181] External Sorter merge with aggregation go to an infinite loop when we have a total ordering

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.3.1
• Fix Version/s: 1.2.3, 1.3.2, 1.4.0
• Component/s: Spark Core
• Assignee: Qiping Li
• Reporter: Qiping Li
• Created: 28/Apr/15 03:52
• Updated: 29/Apr/15 22:53
• Resolved: 29/Apr/15 22:53

Description

In the function mergeWithAggregation of ExternalSorter.scala, when there is a total ordering for keys K, values of the same key in the sorted iterator should be combined. Currently this is done by this:

  val elem = sorted.next()
  val k = elem._1
  var c = elem._2
  while (sorted.hasNext && sorted.head._1 == k) {
    c = mergeCombiners(c, sorted.head._2)
  }

This will go to an infinite loop when there are more than 1 values with the same key. `sorted.next()` should be called to fix this.

Issue Links

• links to: [Github] Pull Request #5737 (chouqin)