[SPARK-837] ResultTask's serialization forget about handling "generation" field, while ShuffleMapTask does

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.7.3, 0.8.0
• Fix Version/s: 0.8.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Andy Huang
• Created: 30/Jul/13 07:29
• Updated: 07/Dec/13 14:37
• Resolved: 09/Sep/13 18:03

Description

In ResultTask's serialization relative method: writeExternal and readExternal, they didn't do anything to generation.

But in ShuffleMapTask's method, writeExternal and readExternal, they do something like "partition = in.readInt()" and " out.writeLong(generation)" to them.

As we know ResultTask will be used after ShuffleMapTask, if right after ShuffleMapTask finish and the work failed for some reason, It will be recomputed, with a "generation" bigger than -1. The ResultTask can't get the right data again with default generation, that it will ask DAGScheduler to recompter ShuffleMapTask again. This will last until the whole job crash.

Issue Links

• duplicates: SPARK-836 ResultTask's serialization forget to handle generation