[SPARK-3690] Closing shuffle writers we swallow more important exception

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.2.0
• Component/s: Spark Core
• Target Version/s: 1.2.0
• Assignee: Unassigned
• Reporter: Egor Pakhomov
• Created: 25/Sep/14 14:41
• Updated: 25/Sep/14 21:50
• Resolved: 25/Sep/14 21:50

Description

ShaffleMapTask: line 75

ShaffleMapTask

 case e: Exception =>
        if (writer != null) {
          writer.stop(success = false)
        }
        throw e

Exception in writer.stop() swallows the important one. Couldn't find the reason for problems for days. Look up in internet "java.io.FileNotFoundException: /local/hd2/yarn/local/usercache/epahomov/appcache/application_1411219858924_12991/spark-local-20140924225309-03f5/21/shuffle_4_12_147 (No such file or directory)" - there are plenty poor guys like me.

Issue Links

• links to: [Github] Pull Request #2537 (epahomov)