[ZEPPELIN-98] Zeppelin does not work on YARN after deps moved from pom.xml to spark/pom.xml

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Assignee: Alexander Bezzubov
• Reporter: Alexander Bezzubov
• Created: 10/Jun/15 05:54
• Updated: 17/Jul/15 16:44
• Resolved: 17/Jul/15 16:44

Description

Zeppelin does not work on YARN after https://github.com/apache/incubator-zeppelin/commit/109b0807fce6453b7543c68069f30b7cece0f20e because of Hadoop version missmatch

in runtime exception looks like

Exception in thread "main" java.lang.AbstractMethodError: org.apache.hadoop.yarn.api.records.ContainerId.setContainerId(J)V
    at org.apache.hadoop.yarn.api.records.ContainerId.newInstance(ContainerId.java:60)
    at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:178)
    at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.getContainerId(YarnSparkHadoopUtil.scala:142)
    at org.apache.spark.deploy.yarn.YarnRMClient.getAttemptId(YarnRMClient.scala:92)
    at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:82)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:587)
...

The reason looks like version Hadoop yarn artefacts version mismatch from below:

Spark 1.4 with Hadoop 2.6

mvn dependency:tree -Pspark-1.4 -Dspark.version=1.4.0-SNAPSHOT -Phadoop-2.6 -Pyarn | grep "org.apache.hadoop:hadoop-yarn-*"
[INFO] |  +- org.apache.hadoop:hadoop-yarn-common:jar:2.4.0:compile
[INFO] |  +- org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:2.4.0:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-yarn-server-common:jar:2.4.0:compile
[INFO] |  \- org.apache.hadoop:hadoop-yarn-client:jar:2.4.0:compile
[INFO] \- org.apache.hadoop:hadoop-yarn-api:jar:2.6.0:compile

Default spark

mvn dependency:tree -Pyarn | grep "org.apache.hadoop:hadoop-yarn-*"                                    
[INFO] |  +- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
[INFO] |  +- org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:2.2.0:compile
[INFO] |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:compile
[INFO] |  \- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
[INFO] \- org.apache.hadoop:hadoop-yarn-api:jar:2.3.0:compile

Spark 1.4

mvn dependency:tree -Pspark-1.4 -Pyarn | grep "org.apache.hadoop:hadoop-yarn-*"

???

Spark 1.2

mvn dependency:tree -Pspark-1.2 -Pyarn | grep "org.apache.hadoop:hadoop-yarn-*"
[INFO] |  +- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:compile
[INFO] |  +- org.apache.hadoop:hadoop-yarn-server-web-proxy:jar:2.2.0:compile
[INFO] |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:compile
[INFO] |  \- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:compile
[INFO] \- org.apache.hadoop:hadoop-yarn-api:jar:2.3.0:compile