[SPARK-6144] When in cluster mode using ADD JAR with a hdfs:// sourced jar will fail

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.3.0
• Component/s: Spark Core
• Target Version/s: 1.3.0
• Assignee: Trystan Leftwich
• Reporter: Trystan Leftwich
• Created: 03/Mar/15 22:29
• Updated: 04/Mar/15 21:03
• Resolved: 04/Mar/15 21:03

Description

While in cluster mode if you use ADD JAR with a HDFS sourced jar it will fail trying to source that jar on the worker nodes with the following error:

15/03/03 04:56:50 ERROR executor.Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.io.FileNotFoundException: /yarn/nm/usercache/vagrant/appcache/application_1425166832391_0027/-19222735701425358546704_cache (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:146)

PR https://github.com/apache/spark/pull/4880

Issue Links

• is broken by: SPARK-4687 SparkContext#addFile doesn't keep file folder information
• links to: [Github] Pull Request #4880 (trystanleftwich)
• links to: [Github] Pull Request #4881 (trystanleftwich)
• links to: [Github] Pull Request #4894 (vanzin)