Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18986

ExternalAppendOnlyMap shouldn't fail when forced to spill before calling its iterator

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.2.0
    • Spark Core
    • None

    Description

      ExternalAppendOnlyMap.forceSpill now uses an assert to check if an iterator is not null in the map. However, the assertion is only true after the map is asked for iterator. Before it, if another memory consumer asks more memory than currently available, ExternalAppendOnlyMap.forceSpill is also be called too. In this case, we will see failure like this:

      [info]   java.lang.AssertionError: assertion failed
      [info]   at scala.Predef$.assert(Predef.scala:156)
      [info]   at org.apache.spark.util.collection.ExternalAppendOnlyMap.forceSpill(ExternalAppendOnlyMap.scala:196)
      [info]   at org.apache.spark.util.collection.Spillable.spill(Spillable.scala:111)
      [info]   at org.apache.spark.util.collection.ExternalAppendOnlyMapSuite$$anonfun$13.apply$mcV$sp(ExternalAppendOnly
      MapSuite.scala:294)
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            viirya L. C. Hsieh
            viirya L. C. Hsieh
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment