[OOZIE-2787] Oozie distributes application jar twice making the spark job fail

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.3.1
• Assignee: Satish Saley
• Reporter: Satish Saley
• Created: 03/Feb/17 05:27
• Updated: 14/Feb/18 13:02
• Resolved: 15/Feb/17 18:42

Description

Oozie adds the application jar to the list of files to be uploaded to distributed cache. Since this gets added twice, the job fails. This is observed from spark 2.1.0 which introduces a check for same file and fails the job.

--master
yarn
--deploy-mode
cluster
--name
oozieSparkStarter
--class
ScalaWordCount
--queue 
default
--conf
spark.executor.extraClassPath=$PWD/*
--conf
spark.driver.extraClassPath=$PWD/*
--conf
spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
--conf
spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
--conf
spark.yarn.security.tokens.hive.enabled=false
--conf
spark.yarn.security.tokens.hbase.enabled=false
--files
hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
--properties-file
spark-defaults.conf
--verbose
spark-example.jar
samplefile.txt
output

Issue Links

• relates to: OOZIE-2554 Spark action tests on yarn-cluster, yarn-client mode