Details
-
Sub-task
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
2.5.0
-
None
-
Reviewed
Description
YarnClient will automatically grab a timeline DT for the application and pass it to the app AM. Now the timeline DT renew is still dummy. If an app is running for more than 24h (default DT expiry time), the app AM is no longer able to use the expired DT to communicate with the timeline server. Since RM will cache the credentials of each app, and renew the DTs for the running app. We should provider renew hooks similar to what HDFS DT has for RM, and set RM user as the renewer when grabbing the timeline DT.
Attachments
Attachments
- YARN-2770.2.patch
- 24 kB
- Zhijie Shen
- YARN-2770.1.patch
- 24 kB
- Zhijie Shen
Issue Links
- is depended upon by
-
YARN-2779 SystemMetricsPublisher can use Kerberos directly instead of timeline DT
- Closed
Activity
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12678042/YARN-2770.1.patch
against trunk revision 0126cf1.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5635//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5635//console
This message is automatically generated.
thanks Zhijie ! some comments:
- SecurityUtil#getServerPrincipal may be useful.
if (rmPrincipal != null && rmPrincipal.length() > 0) { renewer = new KerberosName(rmPrincipal).getServiceName(); }
- We may replace the token after renew is really succeeded.
if (!timelineDT.equals(token.getDelegationToken())) { token.setDelegationToken((Token) timelineDT); }
- In cancelDelegationToken, why replacing the token. Also rename the renewDTAction to cacnelDT
// If the timeline DT to renew is different than cached, replace it. // Token to set every time for retry, because when exception happens, // DelegationTokenAuthenticatedURL will reset it to null; if (!timelineDT.equals(token.getDelegationToken())) { token.setDelegationToken((Token) timelineDT); }
- the same DelegationTokenAuthenticatedURL is instantiated multiple times, is it possible to store it as a variable ?
DelegationTokenAuthenticatedURL authUrl = new DelegationTokenAuthenticatedURL(authenticator, connConfigurator);
- similarly for the timeline client instantiation.
TimelineClient client = TimelineClient.createTimelineClient(); client.init(conf); client.start();
Quick comments:
- Let's make sure the renewer name mangling imitates MR JobClient, it is easy to get this wrong.
- It'll be great to also test separately that renewal can work fine when https is enabled.
SecurityUtil#getServerPrincipal may be useful.
Let's make sure the renewer name mangling imitates MR JobClient, it is easy to get this wrong.
I think we should use HadoopKerberosName#getShortName (AbstractDelegationTokenSecretManager is using it as well) and RM_Principal (which should be there in secure mode) to get the RM daemon user, and HadoopKerberosName will automatically handle auth_to_local if we need to map the auth name to the real operating system name.
It'll be great to also test separately that renewal can work fine when https is enabled.
I've verified it will work with SSL. BTW, SystemMetricsPublisher works fine with SSL too. To make it work, we must make sure RM have seen the proper configuration for SSL and the truststore.
the same DelegationTokenAuthenticatedURL is instantiated multiple times, is it possible to store it as a variable ?
It's probably okay to reuse DelegationTokenAuthenticatedURL. However, I'd like to construct one for each request to isolate the possible resource sharing, preventing introducing potential bugs. Actually Jersey client also construct a new URL for each request. It won't be a big overhead, as it doesn't deeply construct something.
similarly for the timeline client instantiation.
I'm not sure, but guess you're talking about TokenRenewer. Actually I'm following the way that RMDelegationTokenIdentifier does. If we don't construct the client per call, we need to make it a service, and have separate stage for init/start and stop. It may complex the change. Please let me know if you want this change.
We may replace the token after renew is really succeeded.
According to the design of DelegationTokenAuthenticatedURL, I need to put the DT into the current DelegationTokenAuthenticatedURL.Token, which will be fetched internally to do the corresponding operations. So to renew a given DT, I need to set DT there. However, if it already cached there, the client can skip the set step.
Otherwise, I've addressed the remaining comments. Thanks Jian and Vinod!
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12678376/YARN-2770.2.patch
against trunk revision e1f7d65.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. There were no new javadoc warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:
org.apache.hadoop.yarn.client.TestResourceTrackerOnHA
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5650//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5650//console
This message is automatically generated.
The two test failures are not related, and happen on other Jiras, too: file two tickets for them - YARN-2782 an YARN-2783.
FAILURE: Integrated in Hadoop-trunk-Commit #6412 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6412/)
YARN-2770. Added functionality to renew/cancel TimeLineDelegationToken. Contributed by Zhijie Shen (jianhe: rev 1b4be918664b09272b120bc42de3e5fc02d79047)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticatedURL.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineClient.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/TimelineDelegationTokenIdentifier.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/security/TestTimelineAuthenticationFilter.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
SUCCESS: Integrated in Hadoop-Yarn-trunk #730 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/730/)
YARN-2770. Added functionality to renew/cancel TimeLineDelegationToken. Contributed by Zhijie Shen (jianhe: rev 1b4be918664b09272b120bc42de3e5fc02d79047)
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticatedURL.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/TimelineDelegationTokenIdentifier.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineClient.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/security/TestTimelineAuthenticationFilter.java
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1919 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1919/)
YARN-2770. Added functionality to renew/cancel TimeLineDelegationToken. Contributed by Zhijie Shen (jianhe: rev 1b4be918664b09272b120bc42de3e5fc02d79047)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/TimelineDelegationTokenIdentifier.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineClient.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticatedURL.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/security/TestTimelineAuthenticationFilter.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1944 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1944/)
YARN-2770. Added functionality to renew/cancel TimeLineDelegationToken. Contributed by Zhijie Shen (jianhe: rev 1b4be918664b09272b120bc42de3e5fc02d79047)
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticatedURL.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/security/TestTimelineAuthenticationFilter.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineClient.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/TimelineDelegationTokenIdentifier.java
Created a patch: