Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
3.0.0
-
None
Description
BytesToBytesMap's MapIterator's hasNext method is not idempotent.
// public boolean hasNext() { if (numRecords == 0) { if (reader != null) { // if called multiple multiple times, it will throw NoSuchElement exception handleFailedDelete(); } } return numRecords > 0; }
Multiple calls to this `hasNext` method will call `handleFailedDelete()` multiple times, which will throw NoSuchElementException as the spillWrites has already been empty.
We observed this issue for in one of our production jobs after upgrading to Spark 3.0