Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33756

BytesToBytesMap's iterator hasNext method should be idempotent.

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0
    • 2.4.8, 3.0.2, 3.1.0
    • Spark Core
    • None

    Description

      BytesToBytesMap's MapIterator's hasNext method is not idempotent. 

      // 
      public boolean hasNext() {
        if (numRecords == 0) {
          if (reader != null) {
            // if called multiple multiple times, it will throw NoSuchElement exception
            handleFailedDelete();
          }
        }
        return numRecords > 0;
      }
      

      Multiple calls to this `hasNext` method will call `handleFailedDelete()` multiple times, which will throw NoSuchElementException  as the spillWrites has already been empty.

       

      We observed this issue for in one of our production jobs after upgrading to Spark 3.0

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            advancedxy Xianjin YE
            advancedxy Xianjin YE
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment