[SPARK-1124] Infinite NullPointerException failures due to a null in map output locations

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.9.1, 1.0.0
• Component/s: Spark Core
• Assignee: Matei Alexandru Zaharia
• Reporter: Matei Alexandru Zaharia
• Created: 23/Feb/14 19:51
• Updated: 26/Feb/14 09:56
• Resolved: 24/Feb/14 17:04

Description

The following spark-shell code leads to an infinite retry of the last stage in Spark 0.9:

val data = sc.parallelize(1 to 100, 2).map(x => {throw new NullPointerException; (x, x)}).reduceByKey(_ + _)

data.count()    // This first one terminates correctly with just an NPE

data.count()    // This second one never terminates, it keeps failing over and over

The problem seems to be that when there's an NPE in the map stage, we erroneously add map output locations for it, so the next job on the RDD runs only the reduce stage. Those tasks keep failing but they count as a fetch failure, so it keeps retrying.