[SPARK-1031] FileNotFoundException when running simple Spark app on Yarn

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.9.0
• Component/s: Spark Core, YARN
• Assignee: Unassigned
• Reporter: Sanford Ryza
• Created: 19/Jan/14 01:20
• Updated: 20/Jan/14 15:05
• Resolved: 19/Jan/14 21:58

Description

I hit a FileNotFoundException in the application master when running the SparkPi example as described in the docs against Yarn 2.2.

The problem appears to be that the app master requires the app jar to be in its working directory with the same name as returned by SparkContext.jarOfClass. A symlink called app.jar to the app jar exists in the working directory, but jar itself lives in the NodeManager local cache dir with the name it was submitted with (e.g. spark-examples-assembly-0.9.0-incubating-SNAPSHOT.jar). When adding the jar, the SparkContext only uses the last component in its path:

            if (SparkHadoopUtil.get.isYarnMode() && master == "yarn-standalone") {
              // In order for this to work in yarn standalone mode the user must specify the 
              // --addjars option to the client to upload the file into the distributed cache 
              // of the AM to make it show up in the current working directory.
              val fileName = new Path(uri.getPath).getName()
              try {
                env.httpFileServer.addJar(new File(fileName))

If SparkContext.jarOfClass returns the jar in its linked-to location, the file will not be found.

When I modified things to set fileName to use the jar's full path, my app completed successfully.