[OOZIE-3036] Spark 2.2.0 support: tell Spark not to get any delegation tokens

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.3.0
• Fix Version/s: 5.0.0b1, 4.3.1

Environment

• Spark 2.2.0
• Hadoop 3 alpha 4

• Assignee: Andras Piros
• Reporter: Andras Piros
• Created: 14/Aug/17 15:11
• Updated: 25/Jan/18 21:01
• Resolved: 15/Aug/17 09:46

Description

For Spark 2.2.0 along w/ Hadoop 3 alpha 4, Spark uses another set of arguments that ensure Spark won't get any delegation tokens:

• spark.yarn.security.credentials.hadoopfs.enabled
• spark.yarn.security.credentials.hive.enabled
• spark.yarn.security.credentials.hbase.enabled

For Spark 1.6.0 along w/ Hadoop 3 alpha 4, we need following argument to set extra:

• spark.yarn.security.tokens.hadoopfs.enabled

As we can't tell runtime which Spark / Hadoop versions is the Oozie Spark action running on, it's best to use the union of those arguments.