[SPARK-26150] __spark_conf__XXX.zip doesn't exist

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.6.0
• Component/s: Spark Core, YARN
• Labels: bulk-closed
• Flags: Patch
• Assignee: Unassigned
• Reporter: Madhusoodan
• Created: 22/Nov/18 16:46
• Updated: 25/May/21 01:53
• Resolved: 25/May/21 01:41

Description

Spark application submitted via spark-submit fails because the resource localization for the application master instance 2 fails.

 

This is the scenario.

1. I submitted an application via spark-submit
2. The application master failed (The reason is not important here, but here it is: jar was compiled through java 8 and the java installed in cluster nodes was java 7)
3. The appmaster instance 1 deleted the staging directory on failure. (Below is the log excerpt)
   1. Deleting staging directory .sparkStaging/application_1542037527280_0015
4. The Yarn restarted the appmaster in another node, where the NM couldn't find the resource in HDFS resulting in to (from nodemanager logs)
   1. Diagnostics: File does not exist: hdfs://mpcdh001.informatica.com:8020/user/sampleuser/.sparkStaging/application_1542037527280_0007/_spark_conf_7096300976806305459.zip