[SPARK-33756] BytesToBytesMap's iterator hasNext method should be idempotent.

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 2.4.8, 3.0.2, 3.1.0
• Component/s: Spark Core
• Assignee: YE
• Reporter: YE
• Created: 11/Dec/20 09:49
• Updated: 20/Dec/20 14:52
• Resolved: 20/Dec/20 14:52

Description

BytesToBytesMap's MapIterator's hasNext method is not idempotent. 

// 
public boolean hasNext() {
  if (numRecords == 0) {
    if (reader != null) {
      // if called multiple multiple times, it will throw NoSuchElement exception
      handleFailedDelete();
    }
  }
  return numRecords > 0;
}

Multiple calls to this `hasNext` method will call `handleFailedDelete()` multiple times, which will throw NoSuchElementException  as the spillWrites has already been empty.

 

We observed this issue for in one of our production jobs after upgrading to Spark 3.0

Issue Links

• links to: [Github] Pull Request #30728 (advancedxy)