[OOZIE-2787] Oozie distributes application jar twice making the spark job fail - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 4.3.1
Component/s: None
Labels:
None

Description

Oozie adds the application jar to the list of files to be uploaded to distributed cache. Since this gets added twice, the job fails. This is observed from spark 2.1.0 which introduces a check for same file and fails the job.

--master
yarn
--deploy-mode
cluster
--name
oozieSparkStarter
--class
ScalaWordCount
--queue 
default
--conf
spark.executor.extraClassPath=$PWD/*
--conf
spark.driver.extraClassPath=$PWD/*
--conf
spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
--conf
spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
--conf
spark.yarn.security.tokens.hive.enabled=false
--conf
spark.yarn.security.tokens.hbase.enabled=false
--files
hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
--properties-file
spark-defaults.conf
--verbose
spark-example.jar
samplefile.txt
output