[SPARK-4298] The spark-submit cannot read Main-Class from Manifest.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.0.3, 1.1.2, 1.2.1, 1.3.0
• Component/s: Spark Core

Environment

Linux
spark-1.1.0-bin-hadoop2.4.tgz
java version "1.7.0_72"
Java(TM) SE Runtime Environment (build 1.7.0_72-b14)
Java HotSpot(TM) 64-Bit Server VM (build 24.72-b04, mixed mode)

• Assignee: Brennon York
• Reporter: Milan Straka
• Created: 07/Nov/14 14:11
• Updated: 13/Mar/20 09:10
• Resolved: 31/Dec/14 20:02

Description

Consider trivial test.scala:

test.scala

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._

object Main {
  def main(args: Array[String]) {
    val sc = new SparkContext()
    sc.stop()
  }
}

When built with sbt and executed using spark-submit target/scala-2.10/test_2.10-1.0.jar, I get the following error:

Spark assembly has been built with Hive, including Datanucleus jars on classpath
Error: Cannot load main class from JAR: file:/ha/home/straka/s/target/scala-2.10/test_2.10-1.0.jar
Run with --help for usage help or --verbose for debug output

When executed using spark-submit --class Main target/scala-2.10/test_2.10-1.0.jar, it works.

The jar file has correct MANIFEST.MF:

MANIFEST.MF

Manifest-Version: 1.0
Implementation-Vendor: test
Implementation-Title: test
Implementation-Version: 1.0
Implementation-Vendor-Id: test
Specification-Vendor: test
Specification-Title: test
Specification-Version: 1.0
Main-Class: Main

The problem is that in org.apache.spark.deploy.SparkSubmitArguments, line 127:

  val jar = new JarFile(primaryResource)

the primaryResource has String value "file:/ha/home/straka/s/target/scala-2.10/test_2.10-1.0.jar", which is URI, but JarFile can use only Path. One way to fix this would be using

  val uri = new URI(primaryResource)
  val jar = new JarFile(uri.getPath)

Issue Links

• links to: [Github] Pull Request #3561 (brennonyork)