[SPARK-18986] ExternalAppendOnlyMap shouldn't fail when forced to spill before calling its iterator

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: Spark Core
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 23/Dec/16 05:45
• Updated: 11/Jul/17 10:03
• Resolved: 17/Feb/17 19:28

Description

ExternalAppendOnlyMap.forceSpill now uses an assert to check if an iterator is not null in the map. However, the assertion is only true after the map is asked for iterator. Before it, if another memory consumer asks more memory than currently available, ExternalAppendOnlyMap.forceSpill is also be called too. In this case, we will see failure like this:

[info]   java.lang.AssertionError: assertion failed
[info]   at scala.Predef$.assert(Predef.scala:156)
[info]   at org.apache.spark.util.collection.ExternalAppendOnlyMap.forceSpill(ExternalAppendOnlyMap.scala:196)
[info]   at org.apache.spark.util.collection.Spillable.spill(Spillable.scala:111)
[info]   at org.apache.spark.util.collection.ExternalAppendOnlyMapSuite$$anonfun$13.apply$mcV$sp(ExternalAppendOnly
MapSuite.scala:294)

Issue Links

• links to: [Github] Pull Request #16387 (viirya)