[HADOOP-15059] 3.0 deployment cannot work with old version MR tar ball which breaks rolling upgrade - ASF JIRA

Voters

Stop watching

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Blocker
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 3.0.0
Component/s: security
Labels:
None

Target Version/s:

3.0.0
Hadoop Flags:

Reviewed
Release Note:

Hide
This change reverses the default delegation token format implemented by ~~HADOOP-12563~~, but preserves the capability to read the new delegation token format. When the new format becomes default, then MR deployment jobs runs will be compatible with releases that contain this change.

Show
This change reverses the default delegation token format implemented by HADOOP-12563 , but preserves the capability to read the new delegation token format. When the new format becomes default, then MR deployment jobs runs will be compatible with releases that contain this change.

Description

I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed because following error:

2017-11-21 12:42:50,911 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1511295641738_0003_000001
2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-11-21 12:42:51,118 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.RuntimeException: Unable to determine current user
	at org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
	at org.apache.hadoop.conf.Configuration$Resource.<init>(Configuration.java:220)
	at org.apache.hadoop.conf.Configuration$Resource.<init>(Configuration.java:212)
	at org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
Caused by: java.io.IOException: Exception reading /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_000001/container_tokens
	at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
	at org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
	... 4 more
Caused by: java.io.IOException: Unknown version 1 in token storage.
	at org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
	at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
	... 8 more
2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.lang.RuntimeException: Unable to determine current user

I think it is due to token incompatiblity change between 2.9 and 3.0. As we claim "rolling upgrade" is supported in Hadoop 3, we should fix this before we ship 3.0 otherwise all MR running applications will get stuck during/after upgrade.