[SPARK-1667] Jobs never finish successfully once bucket file missing occurred

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Component/s: Shuffle, Spark Core
• Target Version/s: 1.0.3, 1.1.0
• Assignee: Unassigned
• Reporter: Kousuke Saruta
• Created: 29/Apr/14 18:18
• Updated: 17/May/20 18:31
• Resolved: 07/Sep/14 08:14

Description

If jobs execute shuffle, bucket files are created in a temporary directory (named like spark-local-*).
When the bucket files are missing cased by disk failure or any reasons, jobs cannot execute shuffle which has same shuffle id for the bucket files.

Issue Links

• is superceded by: SPARK-2670 FetchFailedException should be thrown when local fetch has failed
• links to: [Github] Pull Request #1383 (sarutak)