Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3227

Timeline renew delegation token fails when RM user's TGT is expired

    Details

    • Target Version/s:

      Description

      When the RM user's kerberos TGT is expired, the RM renew delegation token operation fails as part of job submission. Expected behavior is that RM will relogin to get a new TGT.

      2015-02-06 18:54:05,617 DelegationTokenRenewer #25954 WARN
      security.DelegationTokenRenewer: Unable to add the application to the
      delegation token renewer.
      java.io.IOException: Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN,
      Service: timelineserver.example.com:4080, Ident: (owner=user,
      renewer=rmuser, realUser=oozie, issueDate=1423248845528,
      maxDate=1423853645528, sequenceNumber=9716, masterKeyId=9)
      at
      org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:443)
      at
      org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:77)
      at
      org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:808)
      at
      org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:789)
      at
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:722)
      Caused by: java.io.IOException: HTTP status [401], message [Unauthorized]
      at
      org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:169)
      at
      org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:286)
      at
      org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.renewDelegationToken(DelegationTokenAuthenticator.java:211)
      at
      org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.renewDelegationToken(DelegationTokenAuthenticatedURL.java:414)
      at
      org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$2.run(TimelineClientImpl.java:374)
      at
      org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$2.run(TimelineClientImpl.java:360)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at
      org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1694)
      at
      org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$4.run(TimelineClientImpl.java:429)
      at
      org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:161)
      at
      org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.operateDelegationToken(TimelineClientImpl.java:444)
      at
      org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.renewDelegationToken(TimelineClientImpl.java:378)
      at
      org.apache.hadoop.yarn.security.client.TimelineDelegationTokenIdentifier$Renewer.renew(TimelineDelegationTokenIdentifier.java:81)
      at org.apache.hadoop.security.token.Token.renew(Token.java:377)
      at
      org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:532)
      at
      org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:529)

      1. YARN-3227.1.patch
        1 kB
        Zhijie Shen
      2. YARN-3227.test.patch
        2 kB
        Zhijie Shen

        Activity

        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Is it only the Timeline delegation token that fails renewal or all the tokens?

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Is it only the Timeline delegation token that fails renewal or all the tokens?
        Hide
        zjshen Zhijie Shen added a comment -

        Does HDFS token renew requires the kerberos authentication or just the token authentication? Timeline token requires kerberos authentication being passed first.

        Show
        zjshen Zhijie Shen added a comment - Does HDFS token renew requires the kerberos authentication or just the token authentication? Timeline token requires kerberos authentication being passed first.
        Hide
        zjshen Zhijie Shen added a comment -

        We have encountered the issue too. In addition to renew token operation, publishing system metrics will fail if it is enabled to due to the expired TGT ticket.

        2015-02-11 16:11:57,397 ERROR metrics.SystemMetricsPublisher (SystemMetricsPublisher.java:putEntity(427)) - Error when publishing entity [YARN_APPLICATION,application_1423503166332_0003]
        com.sun.jersey.api.client.ClientHandlerException: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        	at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
        	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineJerseyRetryFilter$1.run(TimelineClientImpl.java:226)
        	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:162)
        	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineJerseyRetryFilter.handle(TimelineClientImpl.java:237)
        	at com.sun.jersey.api.client.Client.handle(Client.java:648)
        	at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
        	at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
        	at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:563)
        	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPostingObject(TimelineClientImpl.java:472)
        	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPosting(TimelineClientImpl.java:321)
        	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:301)
        	at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher.putEntity(SystemMetricsPublisher.java:425)
        	at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher.publishApplicationACLsUpdatedEvent(SystemMetricsPublisher.java:297)
        	at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher.handleSystemMetricsEvent(SystemMetricsPublisher.java:219)
        	at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher$ForwardingEventHandler.handle(SystemMetricsPublisher.java:442)
        	at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher$ForwardingEventHandler.handle(SystemMetricsPublisher.java:437)
        	at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
        	at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
        	at java.lang.Thread.run(Thread.java:745)
        Caused by: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineURLConnectionFactory.getHttpURLConnection(TimelineClientImpl.java:507)
        	at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:159)
        	at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
        	... 18 more
        Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306)
        	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196)
        	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
        	at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
        	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322)
        	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineURLConnectionFactory$1.run(TimelineClientImpl.java:501)
        	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineURLConnectionFactory$1.run(TimelineClientImpl.java:498)
        	at java.security.AccessController.doPrivileged(Native Method)
        	at javax.security.auth.Subject.doAs(Subject.java:415)
        	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineURLConnectionFactory.getHttpURLConnection(TimelineClientImpl.java:498)
        	... 20 more
        Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
        	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
        	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285)
        	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261)
        	at java.security.AccessController.doPrivileged(Native Method)
        	at javax.security.auth.Subject.doAs(Subject.java:415)
        	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261)
        	... 30 more
        

        Compared with WebHdfsFileSystem, timeline client doesn't do checkTGTAndReloginFromKeytab before contacting the timeline server. In RM case, RM user login from the keytab at the beginning, and after 1 day, it may have been already been expired. Then, the operations invoked by RM use may not have the valid TGT ticket to pass Kerberos auth. So we need to add checkTGTAndReloginFromKeytab for both entity/domain and token related operations.

        Show
        zjshen Zhijie Shen added a comment - We have encountered the issue too. In addition to renew token operation, publishing system metrics will fail if it is enabled to due to the expired TGT ticket. 2015-02-11 16:11:57,397 ERROR metrics.SystemMetricsPublisher (SystemMetricsPublisher.java:putEntity(427)) - Error when publishing entity [YARN_APPLICATION,application_1423503166332_0003] com.sun.jersey.api.client.ClientHandlerException: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineJerseyRetryFilter$1.run(TimelineClientImpl.java:226) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:162) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineJerseyRetryFilter.handle(TimelineClientImpl.java:237) at com.sun.jersey.api.client.Client.handle(Client.java:648) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:563) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPostingObject(TimelineClientImpl.java:472) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPosting(TimelineClientImpl.java:321) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:301) at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher.putEntity(SystemMetricsPublisher.java:425) at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher.publishApplicationACLsUpdatedEvent(SystemMetricsPublisher.java:297) at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher.handleSystemMetricsEvent(SystemMetricsPublisher.java:219) at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher$ForwardingEventHandler.handle(SystemMetricsPublisher.java:442) at org.apache.hadoop.yarn.server.resourcemanager.metrics.SystemMetricsPublisher$ForwardingEventHandler.handle(SystemMetricsPublisher.java:437) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106) at java.lang. Thread .run( Thread .java:745) Caused by: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineURLConnectionFactory.getHttpURLConnection(TimelineClientImpl.java:507) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:159) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147) ... 18 more Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127) at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineURLConnectionFactory$1.run(TimelineClientImpl.java:501) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineURLConnectionFactory$1.run(TimelineClientImpl.java:498) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineURLConnectionFactory.getHttpURLConnection(TimelineClientImpl.java:498) ... 20 more Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261) ... 30 more Compared with WebHdfsFileSystem, timeline client doesn't do checkTGTAndReloginFromKeytab before contacting the timeline server. In RM case, RM user login from the keytab at the beginning, and after 1 day, it may have been already been expired. Then, the operations invoked by RM use may not have the valid TGT ticket to pass Kerberos auth. So we need to add checkTGTAndReloginFromKeytab for both entity/domain and token related operations.
        Hide
        zjshen Zhijie Shen added a comment -

        The fix for this problem is straightforward: before starting the connection with the timeline server, the client calls UserGroupInformation#checkTGTAndReloginFromKeytab relogin from the keytab if necessary. I made a patch with 2 lines change for the bug fix.

        It's not easy to verify the problem via the unit test. Instead, I play a trick verify it locally. I shorten the ticket lifetime to 1min, and modify TimelineClientImpl#main to do login from keytab at the beginging, wait for 100s to do token renew, and 100s to put entity. I could reproduce the bug in this way. And after applying the fix, this bug is gone. I attached TimelineClientImpl#main as patch too for your reference.

        Show
        zjshen Zhijie Shen added a comment - The fix for this problem is straightforward: before starting the connection with the timeline server, the client calls UserGroupInformation#checkTGTAndReloginFromKeytab relogin from the keytab if necessary. I made a patch with 2 lines change for the bug fix. It's not easy to verify the problem via the unit test. Instead, I play a trick verify it locally. I shorten the ticket lifetime to 1min, and modify TimelineClientImpl#main to do login from keytab at the beginging, wait for 100s to do token renew, and 100s to put entity. I could reproduce the bug in this way. And after applying the fix, this bug is gone. I attached TimelineClientImpl#main as patch too for your reference.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12703129/YARN-3227.1.patch
        against trunk revision 24db081.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6883//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6883//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12703129/YARN-3227.1.patch against trunk revision 24db081. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6883//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6883//console This message is automatically generated.
        Hide
        xgong Xuan Gong added a comment -

        +1 LGTM. Will commit

        Show
        xgong Xuan Gong added a comment - +1 LGTM. Will commit
        Hide
        jeagles Jonathan Eagles added a comment -

        Xuan Gong, please let me have a chance at a second review as the reporter of this issue.

        Show
        jeagles Jonathan Eagles added a comment - Xuan Gong , please let me have a chance at a second review as the reporter of this issue.
        Hide
        xgong Xuan Gong added a comment -

        Committed to trunk/branch-2. Thanks, zhijie

        Show
        xgong Xuan Gong added a comment - Committed to trunk/branch-2. Thanks, zhijie
        Hide
        xgong Xuan Gong added a comment -

        Jonathan Eagles Sorry, I missed your message. Please review it. If the patch does not make sense to you, feel free to re-open it.

        Show
        xgong Xuan Gong added a comment - Jonathan Eagles Sorry, I missed your message. Please review it. If the patch does not make sense to you, feel free to re-open it.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7272 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7272/)
        YARN-3227. Timeline renew delegation token fails when RM user's TGT is (xgong: rev d1abc5d4fc00bb1b226066684556ba16ace71744)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7272 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7272/ ) YARN-3227 . Timeline renew delegation token fails when RM user's TGT is (xgong: rev d1abc5d4fc00bb1b226066684556ba16ace71744) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java hadoop-yarn-project/CHANGES.txt
        Hide
        jeagles Jonathan Eagles added a comment -

        Xuan Gong, as a matter of future reference, please give the active participants a chance to continue dialog for a newly posted patch. A +1 and commit from a non-active participant minutes after the +1 from HadooQA is considered impolite.

        Show
        jeagles Jonathan Eagles added a comment - Xuan Gong , as a matter of future reference, please give the active participants a chance to continue dialog for a newly posted patch. A +1 and commit from a non-active participant minutes after the +1 from HadooQA is considered impolite.
        Hide
        zjshen Zhijie Shen added a comment -

        Hi Jon, sorry for being hurry about this bug as it's critical to us. Please feel free to reopen it if you have any concern about it.

        Show
        zjshen Zhijie Shen added a comment - Hi Jon, sorry for being hurry about this bug as it's critical to us. Please feel free to reopen it if you have any concern about it.
        Hide
        jeagles Jonathan Eagles added a comment -

        +1. Patch looks good. Is there no test to be checked in?

        Show
        jeagles Jonathan Eagles added a comment - +1. Patch looks good. Is there no test to be checked in?
        Hide
        zjshen Zhijie Shen added a comment -

        Yup, I didn't figure out a quick way to simulate the TGT ticket expiry in the unit test. If I missed something obvious, please let me know. We can definitely add one test case to cover the change.

        On the other hand, to verify if the code change will work, I made a small change in TimelineClientImpl#main and changed my local krb5.conf to shorten the ticket lifetime. I could reproduce the problem and get rid of it by applying the fix.

        Show
        zjshen Zhijie Shen added a comment - Yup, I didn't figure out a quick way to simulate the TGT ticket expiry in the unit test. If I missed something obvious, please let me know. We can definitely add one test case to cover the change. On the other hand, to verify if the code change will work, I made a small change in TimelineClientImpl#main and changed my local krb5.conf to shorten the ticket lifetime. I could reproduce the problem and get rid of it by applying the fix.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #125 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/125/)
        YARN-3227. Timeline renew delegation token fails when RM user's TGT is (xgong: rev d1abc5d4fc00bb1b226066684556ba16ace71744)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #125 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/125/ ) YARN-3227 . Timeline renew delegation token fails when RM user's TGT is (xgong: rev d1abc5d4fc00bb1b226066684556ba16ace71744) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #859 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/859/)
        YARN-3227. Timeline renew delegation token fails when RM user's TGT is (xgong: rev d1abc5d4fc00bb1b226066684556ba16ace71744)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #859 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/859/ ) YARN-3227 . Timeline renew delegation token fails when RM user's TGT is (xgong: rev d1abc5d4fc00bb1b226066684556ba16ace71744) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2057 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2057/)
        YARN-3227. Timeline renew delegation token fails when RM user's TGT is (xgong: rev d1abc5d4fc00bb1b226066684556ba16ace71744)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2057 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2057/ ) YARN-3227 . Timeline renew delegation token fails when RM user's TGT is (xgong: rev d1abc5d4fc00bb1b226066684556ba16ace71744) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #116 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/116/)
        YARN-3227. Timeline renew delegation token fails when RM user's TGT is (xgong: rev d1abc5d4fc00bb1b226066684556ba16ace71744)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #116 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/116/ ) YARN-3227 . Timeline renew delegation token fails when RM user's TGT is (xgong: rev d1abc5d4fc00bb1b226066684556ba16ace71744) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #125 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/125/)
        YARN-3227. Timeline renew delegation token fails when RM user's TGT is (xgong: rev d1abc5d4fc00bb1b226066684556ba16ace71744)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #125 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/125/ ) YARN-3227 . Timeline renew delegation token fails when RM user's TGT is (xgong: rev d1abc5d4fc00bb1b226066684556ba16ace71744) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2075 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2075/)
        YARN-3227. Timeline renew delegation token fails when RM user's TGT is (xgong: rev d1abc5d4fc00bb1b226066684556ba16ace71744)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2075 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2075/ ) YARN-3227 . Timeline renew delegation token fails when RM user's TGT is (xgong: rev d1abc5d4fc00bb1b226066684556ba16ace71744) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java hadoop-yarn-project/CHANGES.txt
        Hide
        alexmc Alex McLintock added a comment -

        Are there any work arounds for this that I can use until I can get a version of Hadoop with this patch in it?

        Show
        alexmc Alex McLintock added a comment - Are there any work arounds for this that I can use until I can get a version of Hadoop with this patch in it?
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Pulled this into 2.6.1. Ran compilation before the push. Patch applied cleanly.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Pulled this into 2.6.1. Ran compilation before the push. Patch applied cleanly.

          People

          • Assignee:
            zjshen Zhijie Shen
            Reporter:
            jeagles Jonathan Eagles
          • Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development