[HADOOP-15059] 3.0 deployment cannot work with old version MR tar ball which breaks rolling upgrade

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: security
• Target Version/s: 3.0.0
• Hadoop Flags: Reviewed

Release Note

This change reverses the default delegation token format implemented by HADOOP-12563, but preserves the capability to read the new delegation token format. When the new format becomes default, then MR deployment jobs runs will be compatible with releases that contain this change.

• Assignee: Jason Darrell Lowe
• Reporter: Junping Du
• Created: 21/Nov/17 20:54
• Updated: 18/Dec/19 18:26
• Resolved: 08/Dec/17 16:05

Description

I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed because following error:

2017-11-21 12:42:50,911 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1511295641738_0003_000001
2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-11-21 12:42:51,118 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.RuntimeException: Unable to determine current user
	at org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
	at org.apache.hadoop.conf.Configuration$Resource.<init>(Configuration.java:220)
	at org.apache.hadoop.conf.Configuration$Resource.<init>(Configuration.java:212)
	at org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
Caused by: java.io.IOException: Exception reading /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_000001/container_tokens
	at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
	at org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
	... 4 more
Caused by: java.io.IOException: Unknown version 1 in token storage.
	at org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
	at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
	... 8 more
2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.lang.RuntimeException: Unable to determine current user

I think it is due to token incompatiblity change between 2.9 and 3.0. As we claim "rolling upgrade" is supported in Hadoop 3, we should fix this before we ship 3.0 otherwise all MR running applications will get stuck during/after upgrade.

Issue Links

• is broken by: HADOOP-12563 Updated utility to create/modify token files
• is related to: HDFS-12920 HDFS default value change (with adding time unit) breaks old version MR tarball work with Hadoop 3.x
• relates to: HADOOP-16621 [pb-upgrade] Remove Protobuf classes from signatures of Public APIs
• relates to: HADOOP-13123 Permit the default hadoop delegation token file format to be configurable
• relates to: HADOOP-15501 [Umbrella] Upgrade efforts to Hadoop 3.x