Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33756

BytesToBytesMap's iterator hasNext method should be idempotent.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 2.4.8, 3.0.2, 3.1.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      BytesToBytesMap's MapIterator's hasNext method is not idempotent. 

      // 
      public boolean hasNext() {
        if (numRecords == 0) {
          if (reader != null) {
            // if called multiple multiple times, it will throw NoSuchElement exception
            handleFailedDelete();
          }
        }
        return numRecords > 0;
      }
      

      Multiple calls to this `hasNext` method will call `handleFailedDelete()` multiple times, which will throw NoSuchElementException  as the spillWrites has already been empty.

       

      We observed this issue for in one of our production jobs after upgrading to Spark 3.0

        Attachments

          Activity

            People

            • Assignee:
              advancedxy Xianjin YE
              Reporter:
              advancedxy Xianjin YE
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: