Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
    • Fix Version/s: 2.9.0, 2.7.4, 2.6.6, 3.0.0-alpha4, 2.8.2
    • Component/s: security
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      This is a problem that has troubled us for several years. For our HBase cluster, sometimes the RS will be stuck due to

      2016-06-20,03:44:12,936 INFO org.apache.hadoop.ipc.SecureClient: Exception encountered while connecting to the server :
      javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: The ticket isn't for us (35) - BAD TGS SERVER NAME)]
              at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
              at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:140)
              at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:187)
              at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$700(SecureClient.java:95)
              at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:325)
              at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:322)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:396)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1781)
              at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
              at org.apache.hadoop.hbase.security.User.call(User.java:607)
              at org.apache.hadoop.hbase.security.User.access$700(User.java:51)
              at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:461)
              at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:321)
              at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1164)
              at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1004)
              at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:107)
              at $Proxy24.replicateLogEntries(Unknown Source)
              at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:962)
              at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.runLoop(ReplicationSource.java:466)
              at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:515)
      Caused by: GSSException: No valid credentials provided (Mechanism level: The ticket isn't for us (35) - BAD TGS SERVER NAME)
              at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:663)
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:180)
              at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
              ... 23 more
      Caused by: KrbException: The ticket isn't for us (35) - BAD TGS SERVER NAME
              at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:64)
              at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:185)
              at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:294)
              at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:106)
              at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:557)
              at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:594)
              ... 26 more
      Caused by: KrbException: Identifier doesn't match expected value (906)
              at sun.security.krb5.internal.KDCRep.init(KDCRep.java:133)
              at sun.security.krb5.internal.TGSRep.init(TGSRep.java:58)
              at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:53)
              at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:46)
              ... 31 more​
      

      It rarely happens, but if it happens, the regionserver will be stuck and can never recover.

      Recently we added a log after a successful re-login which prints the private credentials, and finally catched the direct reason. After a successful re-login, we have two kerberos tickets in the credentials, one is the TGT, and the other is a service ticket. The strange thing is that, the service ticket is placed before TGT. This breaks the assumption of jdk's kerberos library. See http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/file/935758609767/src/share/classes/sun/security/jgss/krb5/Krb5InitCredential.java, the getTgt Method

      Krb5InitCredential
                  return AccessController.doPrivileged(
                      new PrivilegedExceptionAction<KerberosTicket>() {
                      public KerberosTicket run() throws Exception {
                          // It's OK to use null as serverPrincipal. TGT is almost
                          // the first ticket for a principal and we use list.
                          return Krb5Util.getTicket(
                              realCaller,
                              clientPrincipal, null, acc);
                              }});
      

      So here, the library will use the service ticket as TGT to acquire a service ticket, and KDC will reject the request since the 'TGT' does not start with 'krbtgt'. And it can never recover because in UGI, the re-login will check if there is a valid TGT first and no doubt, we have one...

      This usually happens when a secure connection initialization comes along with the re-login, and the end time indicates that the service ticket is acquired by the previous TGT. Since UGI does not prevent doAs and re-login happen at the same time, we believe that there is a race condition.

      After reading the code, we found a possible race condition.

      See http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/file/935758609767/src/share/classes/sun/security/jgss/krb5/Krb5Context.java, the initSecContext method, we will get TGT first, then check if there is already a service ticket, if not, acquire a service ticket using the TGT, and put it into the credentials.

      And in Krb5LoginModule.logout(the sun version), we will remove the kerberos tickets from the credentials first, and then destroy them.

      Here comes the race condition. Let T1 be the secure connection set up thread, T2 be the re-login thread.

      T1: get TGT
      T2: remove all tickets from credentials
      T1: check service ticket, none(since all tickets have been removed)
      T1: acquire a new service ticket using TGT and put it into the credentials
      T2: destroy all tickets
      T2: login, i.e., put a new TGT into the credentials.

      It is hard to write a UT to produce the problem because the racing code is in jdk, which is not written by us...

      Suggestions are welcomed. Thanks.

      1. HADOOP-13433.patch
        13 kB
        Duo Zhang
      2. HADOOP-13433-v1.patch
        13 kB
        Duo Zhang
      3. HADOOP-13433-v2.patch
        13 kB
        Duo Zhang
      4. HBASE-13433-testcase-v3.patch
        7 kB
        Duo Zhang
      5. HADOOP-13433-v4.patch
        20 kB
        Duo Zhang
      6. HADOOP-13433-v5.patch
        19 kB
        Duo Zhang
      7. HADOOP-13433-v6.patch
        19 kB
        Duo Zhang
      8. HADOOP-13433-branch-2.patch
        6 kB
        Duo Zhang
      9. HADOOP-13433-branch-2.8.patch
        6 kB
        Duo Zhang
      10. HADOOP-13433-branch-2.7.patch
        6 kB
        Duo Zhang
      11. HADOOP-13433-branch-2.8.patch
        6 kB
        Xiao Chen
      12. HADOOP-13433-branch-2.7-v1.patch
        6 kB
        Duo Zhang
      13. HADOOP-13433-branch-2.8-v1.patch
        6 kB
        Duo Zhang
      14. HADOOP-13433-branch-2.7-v2.patch
        6 kB
        Duo Zhang

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          sounds like there needs to be some exclusions so that you can't mix doAs and (re) logins, though there's always the risk of deadlocks coming there -or at least relogin not happening due to the lifespan of doAs work

          Show
          stevel@apache.org Steve Loughran added a comment - sounds like there needs to be some exclusions so that you can't mix doAs and (re) logins, though there's always the risk of deadlocks coming there -or at least relogin not happening due to the lifespan of doAs work
          Hide
          Apache9 Duo Zhang added a comment -

          In general, a better way is to not reuse the Subject object. The re-login stages will be

          1. Create a new subject and login.
          2. Switch subject.
          3. logout the old subject some while later since it may still be used by someone.

          I used to write code like this, it works. But I do not know if it works in hadoop. If we add lots of other stuffs to the subject, then the algorithm maybe broken because we do not know when to move these stuffs from the old subject to new subject.

          Thanks.

          Show
          Apache9 Duo Zhang added a comment - In general, a better way is to not reuse the Subject object. The re-login stages will be 1. Create a new subject and login. 2. Switch subject. 3. logout the old subject some while later since it may still be used by someone. I used to write code like this, it works. But I do not know if it works in hadoop. If we add lots of other stuffs to the subject, then the algorithm maybe broken because we do not know when to move these stuffs from the old subject to new subject. Thanks.
          Hide
          Apache9 Duo Zhang added a comment -

          Some progress...

          I tried to write a UT by moving TGT to the last of the private credentials manually, and the service ticket is sent to KDC as expected when creating a SaslClient. But our MiniKdc does not check the prefix of a TGT so there is no error...

          Show
          Apache9 Duo Zhang added a comment - Some progress... I tried to write a UT by moving TGT to the last of the private credentials manually, and the service ticket is sent to KDC as expected when creating a SaslClient. But our MiniKdc does not check the prefix of a TGT so there is no error...
          Hide
          Apache9 Duo Zhang added a comment -

          Let's fix the bug first. I found that the minikdc on trunk could reject a tgs request which uses a service ticket as TGT.

          The difficulty of changing the relogin steps is that, the subject of UserGroupInformation can be passed from outside and UserGroupInformation is declared as LimitedPrivate, so it is not safe to switch subject after relogin. We need to discuss with the projects declared in LimitedPrivate annotations if we want to do this. I can file another issue to do this.

          Thanks.

          Show
          Apache9 Duo Zhang added a comment - Let's fix the bug first. I found that the minikdc on trunk could reject a tgs request which uses a service ticket as TGT. The difficulty of changing the relogin steps is that, the subject of UserGroupInformation can be passed from outside and UserGroupInformation is declared as LimitedPrivate, so it is not safe to switch subject after relogin. We need to discuss with the projects declared in LimitedPrivate annotations if we want to do this. I can file another issue to do this. Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 47s trunk passed
          +1 compile 6m 49s trunk passed
          +1 checkstyle 0m 24s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 18s trunk passed
          +1 javadoc 0m 45s trunk passed
          +1 mvninstall 0m 37s the patch passed
          +1 compile 6m 42s the patch passed
          +1 javac 6m 43s the patch passed
          -0 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 27 new + 96 unchanged - 2 fixed = 123 total (was 98)
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 1m 26s the patch passed
          +1 javadoc 0m 46s the patch passed
          +1 unit 8m 14s hadoop-common in the patch passed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          38m 27s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825021/HADOOP-13433.patch
          JIRA Issue HADOOP-13433
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 49d5518b9c9f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0d5997d
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10343/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10343/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10343/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 47s trunk passed +1 compile 6m 49s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 18s trunk passed +1 javadoc 0m 45s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 6m 42s the patch passed +1 javac 6m 43s the patch passed -0 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 27 new + 96 unchanged - 2 fixed = 123 total (was 98) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 1m 26s the patch passed +1 javadoc 0m 46s the patch passed +1 unit 8m 14s hadoop-common in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 38m 27s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825021/HADOOP-13433.patch JIRA Issue HADOOP-13433 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 49d5518b9c9f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0d5997d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10343/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10343/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10343/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          Apache9 Duo Zhang added a comment -

          checkstyle.

          Show
          Apache9 Duo Zhang added a comment - checkstyle.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 15s trunk passed
          +1 compile 8m 8s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 1m 3s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 23s trunk passed
          +1 javadoc 0m 44s trunk passed
          +1 mvninstall 0m 38s the patch passed
          +1 compile 6m 44s the patch passed
          +1 javac 6m 44s the patch passed
          -0 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 1 new + 96 unchanged - 2 fixed = 97 total (was 98)
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 45s the patch passed
          +1 javadoc 0m 48s the patch passed
          +1 unit 9m 17s hadoop-common in the patch passed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          43m 10s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825196/HADOOP-13433-v1.patch
          JIRA Issue HADOOP-13433
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux b6ddca51a2a4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c37346d
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10351/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10351/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10351/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 15s trunk passed +1 compile 8m 8s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 23s trunk passed +1 javadoc 0m 44s trunk passed +1 mvninstall 0m 38s the patch passed +1 compile 6m 44s the patch passed +1 javac 6m 44s the patch passed -0 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 1 new + 96 unchanged - 2 fixed = 97 total (was 98) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 48s the patch passed +1 unit 9m 17s hadoop-common in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 43m 10s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825196/HADOOP-13433-v1.patch JIRA Issue HADOOP-13433 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux b6ddca51a2a4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c37346d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10351/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10351/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10351/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          Apache9 Duo Zhang added a comment -

          checkstyle...

          Show
          Apache9 Duo Zhang added a comment - checkstyle...
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 44s trunk passed
          +1 compile 6m 47s trunk passed
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 0m 53s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 18s trunk passed
          +1 javadoc 0m 45s trunk passed
          +1 mvninstall 0m 37s the patch passed
          +1 compile 6m 42s the patch passed
          +1 javac 6m 42s the patch passed
          +1 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 0 new + 96 unchanged - 2 fixed = 96 total (was 98)
          +1 mvnsite 0m 52s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 26s the patch passed
          +1 javadoc 0m 46s the patch passed
          -1 unit 17m 2s hadoop-common in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          47m 10s



          Reason Tests
          Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825208/HADOOP-13433-v2.patch
          JIRA Issue HADOOP-13433
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux cf58bceb601d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c37346d
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10355/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10355/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10355/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 44s trunk passed +1 compile 6m 47s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 18s trunk passed +1 javadoc 0m 45s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 6m 42s the patch passed +1 javac 6m 42s the patch passed +1 checkstyle 0m 24s hadoop-common-project/hadoop-common: The patch generated 0 new + 96 unchanged - 2 fixed = 96 total (was 98) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 26s the patch passed +1 javadoc 0m 46s the patch passed -1 unit 17m 2s hadoop-common in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 47m 10s Reason Tests Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825208/HADOOP-13433-v2.patch JIRA Issue HADOOP-13433 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux cf58bceb601d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c37346d Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10355/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10355/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10355/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          Apache9 Duo Zhang added a comment -

          https://builds.apache.org/job/PreCommit-HADOOP-Build/10355/

          The result is green... I do not know why the QA bot said TestHttpServerLifecycle timed out...

          Show
          Apache9 Duo Zhang added a comment - https://builds.apache.org/job/PreCommit-HADOOP-Build/10355/ The result is green... I do not know why the QA bot said TestHttpServerLifecycle timed out...
          Hide
          Apache9 Duo Zhang added a comment -

          Any comments? Thanks.

          Show
          Apache9 Duo Zhang added a comment - Any comments? Thanks.
          Hide
          Apache9 Duo Zhang added a comment -

          Some progress here. Now I can reproduce the bug with the UT in patch v3. Steve Loughran Could you please try it locally to see if it also works for you? And is it enough to say that the bug can happen in the real world and we need to fix it?

          Thanks.

          Show
          Apache9 Duo Zhang added a comment - Some progress here. Now I can reproduce the bug with the UT in patch v3. Steve Loughran Could you please try it locally to see if it also works for you? And is it enough to say that the bug can happen in the real world and we need to fix it? Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 29s trunk passed
          +1 compile 9m 47s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 1m 22s trunk passed
          +1 javadoc 0m 47s trunk passed
          +1 mvninstall 0m 36s the patch passed
          +1 compile 9m 9s the patch passed
          +1 javac 9m 9s the patch passed
          -0 checkstyle 0m 27s hadoop-common-project/hadoop-common: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 mvnsite 0m 59s the patch passed
          +1 mvneclipse 0m 18s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 30s the patch passed
          +1 javadoc 0m 47s the patch passed
          -1 unit 8m 5s hadoop-common in the patch failed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          50m 34s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13433
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845021/HBASE-13433-testcase-v3.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5e6724047579 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 603f3ef
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11332/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11332/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11332/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11332/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 29s trunk passed +1 compile 9m 47s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 22s trunk passed +1 javadoc 0m 47s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 9m 9s the patch passed +1 javac 9m 9s the patch passed -0 checkstyle 0m 27s hadoop-common-project/hadoop-common: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 mvnsite 0m 59s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 30s the patch passed +1 javadoc 0m 47s the patch passed -1 unit 8m 5s hadoop-common in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 50m 34s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13433 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845021/HBASE-13433-testcase-v3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5e6724047579 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 603f3ef Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11332/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11332/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11332/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11332/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stack stack added a comment -

          Should the test case be integrated into the patch Duo Zhang?

          Have you deployed this fix on your clusters?

          Patch LGTM.

          Any opinion mighty Steve Loughran?

          Show
          stack stack added a comment - Should the test case be integrated into the patch Duo Zhang ? Have you deployed this fix on your clusters? Patch LGTM. Any opinion mighty Steve Loughran ?
          Hide
          Apache9 Duo Zhang added a comment -

          Have you deployed this fix on your clusters?

          Yes, we have been running several clusters with this patch in for about half a year. It works fine.

          Should the test case be integrated into the patch

          Sure. Let me prepare a new patch.

          Show
          Apache9 Duo Zhang added a comment - Have you deployed this fix on your clusters? Yes, we have been running several clusters with this patch in for about half a year. It works fine. Should the test case be integrated into the patch Sure. Let me prepare a new patch.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Great analysis and patch, Duo Zhang!

          The fix looks good to me. However it needs to rebase.

          I glanced through the v2 and v3 test cases and both looks good to me, although it would be awesome if you could remove the wildcard import

          import static org.junit.Assert.*;

          Steve Loughran is it okay to make hadoop-common depend on hadoop-minikdc for tests? I think it's fine but want to double check to avoid unwanted side effects.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Great analysis and patch, Duo Zhang ! The fix looks good to me. However it needs to rebase. I glanced through the v2 and v3 test cases and both looks good to me, although it would be awesome if you could remove the wildcard import import static org.junit.Assert.*; Steve Loughran is it okay to make hadoop-common depend on hadoop-minikdc for tests? I think it's fine but want to double check to avoid unwanted side effects.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          It might be better to move a test into minikdc for consistency

          We allow .* wildcard imports of static methods; they're pretty common in a lot of test cases (especially the ones I write)

          Show
          stevel@apache.org Steve Loughran added a comment - It might be better to move a test into minikdc for consistency We allow .* wildcard imports of static methods; they're pretty common in a lot of test cases (especially the ones I write)
          Hide
          stevel@apache.org Steve Loughran added a comment -

          BTW, we need an affects-version marker on the issue

          Show
          stevel@apache.org Steve Loughran added a comment - BTW, we need an affects-version marker on the issue
          Hide
          Apache9 Duo Zhang added a comment -

          I do not think we can put all the tests in minikdc as we need to test the logic of UserGroupInformation...
          Or is it better to put the tests in a sub-module that depends both on hadoop-common and minikdc already?

          Thanks.

          Show
          Apache9 Duo Zhang added a comment - I do not think we can put all the tests in minikdc as we need to test the logic of UserGroupInformation... Or is it better to put the tests in a sub-module that depends both on hadoop-common and minikdc already? Thanks.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          looks like Hadoop already has minikdc as a test-time dependency, So no issues with usng it here.

          Show
          stevel@apache.org Steve Loughran added a comment - looks like Hadoop already has minikdc as a test-time dependency, So no issues with usng it here.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          OK, real review Review

          core patch looks good.

          • patches need to be combined to one; and yetus be happy
          • UserGroupInformation uses SL4J, so logging should be done with ("ticket server is {}", ticketServer) code style
          • I don't like the "is this possible?" warning. Better to be blunt: "Warning, no kerberos ticket found while attempting to renew ticket". I'm sure that the condition could be reached if someone tried hard enough (i.e. used kdestroy).
          • Tests should consider LambdaTestUtils.intercept for exception handling, maybe look for the exact exception expected
          • sleep time of 10ms is a bit short & would stop everything else, how about 50 or 100ms?
          • assertTrue/assertFalse in assertions need some details on why they are failing, so that people looking at jenkins test logs have a hint as to what went wrong.
          Show
          stevel@apache.org Steve Loughran added a comment - OK, real review Review core patch looks good. patches need to be combined to one; and yetus be happy UserGroupInformation uses SL4J, so logging should be done with ("ticket server is {}", ticketServer) code style I don't like the "is this possible?" warning. Better to be blunt: "Warning, no kerberos ticket found while attempting to renew ticket". I'm sure that the condition could be reached if someone tried hard enough (i.e. used kdestroy). Tests should consider LambdaTestUtils.intercept for exception handling, maybe look for the exact exception expected sleep time of 10ms is a bit short & would stop everything else, how about 50 or 100ms? assertTrue/assertFalse in assertions need some details on why they are failing, so that people looking at jenkins test logs have a hint as to what went wrong.
          Hide
          hexiaoqiao He Xiaoqiao added a comment -

          Thanks Duo Zhang for you contribution. it's very useful patch and i met the same problem recently, my question is when i patch to branch-2.7.1, UT(TestFixKerberosTicketOrder) can't pass, any suggestion? thanks again.

          Show
          hexiaoqiao He Xiaoqiao added a comment - Thanks Duo Zhang for you contribution. it's very useful patch and i met the same problem recently, my question is when i patch to branch-2.7.1, UT(TestFixKerberosTicketOrder) can't pass, any suggestion? thanks again.
          Hide
          Apache9 Duo Zhang added a comment -

          Address the review comments. And use lambda expression in UTs as trunk is jdk8 only? Can rewrite the UTs when back porting.

          Show
          Apache9 Duo Zhang added a comment - Address the review comments. And use lambda expression in UTs as trunk is jdk8 only? Can rewrite the UTs when back porting.
          Hide
          Apache9 Duo Zhang added a comment -

          What's the problem?

          In our company we are still running 2.4.x and 2.6.x, so I haven't tested it for 2.7.x yet.

          Show
          Apache9 Duo Zhang added a comment - What's the problem? In our company we are still running 2.4.x and 2.6.x, so I haven't tested it for 2.7.x yet.
          Hide
          hexiaoqiao He Xiaoqiao added a comment - - edited

          Duo Zhang,Thank for your comments.
          when i patch v2 to branch-2.7.*, and run TestFixKerberosTicketOrder, it throw AssertionError as following:

          Running org.apache.hadoop.security.TestRaceWhenRelogin
          Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.18 sec <<< FAILURE! - in org.apache.hadoop.security.TestRaceWhenRelogin
          test(org.apache.hadoop.security.TestRaceWhenRelogin)  Time elapsed: 4.001 sec  <<< FAILURE!
          java.lang.AssertionError: Should fail because we send a server ticket as TGT
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.apache.hadoop.security.TestRaceWhenRelogin.test(TestRaceWhenRelogin.java:178)
          

          after review code, found the following code always run, is it necessary?

                fail("Should fail because we send a server ticket as TGT");
          

          BTW, configuration item CommonConfigurationKeys.HADOOP_KERBEROS_MIN_SECONDS_BEFORE_RELOGIN seems never be used, why set value?

          pls correct me if i am wrong.

          Show
          hexiaoqiao He Xiaoqiao added a comment - - edited Duo Zhang ,Thank for your comments. when i patch v2 to branch-2.7.*, and run TestFixKerberosTicketOrder , it throw AssertionError as following: Running org.apache.hadoop.security.TestRaceWhenRelogin Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.18 sec <<< FAILURE! - in org.apache.hadoop.security.TestRaceWhenRelogin test(org.apache.hadoop.security.TestRaceWhenRelogin) Time elapsed: 4.001 sec <<< FAILURE! java.lang.AssertionError: Should fail because we send a server ticket as TGT at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.security.TestRaceWhenRelogin.test(TestRaceWhenRelogin.java:178) after review code, found the following code always run, is it necessary? fail( "Should fail because we send a server ticket as TGT" ); BTW, configuration item CommonConfigurationKeys.HADOOP_KERBEROS_MIN_SECONDS_BEFORE_RELOGIN seems never be used, why set value? pls correct me if i am wrong.
          Hide
          Apache9 Duo Zhang added a comment -

          Oh, TestRaceWhenRelogin. The code I put here is used to verify that the problem can happen without moving the tickets manually, so TestRaceWhenRelogin will pass without the fix...

          In the new patch it is used to verify that tgt will always be the first ticket after relogin.

          And for CommonConfigurationKeys.HADOOP_KERBEROS_MIN_SECONDS_BEFORE_RELOGIN, yeah we have shouldRenewImmediatelyForTests now so we do not need to set it anymore.

          Thanks.

          Show
          Apache9 Duo Zhang added a comment - Oh, TestRaceWhenRelogin. The code I put here is used to verify that the problem can happen without moving the tickets manually, so TestRaceWhenRelogin will pass without the fix... In the new patch it is used to verify that tgt will always be the first ticket after relogin. And for CommonConfigurationKeys.HADOOP_KERBEROS_MIN_SECONDS_BEFORE_RELOGIN , yeah we have shouldRenewImmediatelyForTests now so we do not need to set it anymore. Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 14m 20s trunk passed
          +1 compile 10m 13s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 2s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 1m 25s trunk passed
          +1 javadoc 0m 47s trunk passed
          +1 mvninstall 0m 36s the patch passed
          +1 compile 9m 6s the patch passed
          +1 javac 9m 6s the patch passed
          -0 checkstyle 0m 30s hadoop-common-project/hadoop-common: The patch generated 3 new + 94 unchanged - 2 fixed = 97 total (was 96)
          +1 mvnsite 0m 58s the patch passed
          +1 mvneclipse 0m 18s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 29s the patch passed
          +1 javadoc 0m 48s the patch passed
          -1 unit 8m 5s hadoop-common in the patch failed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          52m 59s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13433
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846745/HADOOP-13433-v4.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux be7d50c756ba 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 467f5f1
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11414/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11414/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11414/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11414/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 14m 20s trunk passed +1 compile 10m 13s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 25s trunk passed +1 javadoc 0m 47s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 9m 6s the patch passed +1 javac 9m 6s the patch passed -0 checkstyle 0m 30s hadoop-common-project/hadoop-common: The patch generated 3 new + 94 unchanged - 2 fixed = 97 total (was 96) +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 29s the patch passed +1 javadoc 0m 48s the patch passed -1 unit 8m 5s hadoop-common in the patch failed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 52m 59s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13433 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846745/HADOOP-13433-v4.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux be7d50c756ba 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 467f5f1 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11414/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11414/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11414/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11414/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          Apache9 Duo Zhang added a comment -

          The failed UT seems unrelated and can pass locally.

          Show
          Apache9 Duo Zhang added a comment - The failed UT seems unrelated and can pass locally.
          Hide
          Apache9 Duo Zhang added a comment -
          Show
          Apache9 Duo Zhang added a comment - Ping Steve Loughran , Wei-Chiu Chuang . Thanks.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Duo Zhang thanks for reporting this issue and working on this..

          we've got same issue from production cluster..your approach looks good to me..can you upload branch-2.7 patch also..?

          pinging Steve Loughran and Wei-Chiu Chuang

          Show
          brahmareddy Brahma Reddy Battula added a comment - Duo Zhang thanks for reporting this issue and working on this.. we've got same issue from production cluster..your approach looks good to me..can you upload branch-2.7 patch also..? pinging Steve Loughran and Wei-Chiu Chuang
          Hide
          Apache9 Duo Zhang added a comment -

          can you upload branch-2.7 patch also..?

          Sure. Will upload later. I think the problem is the UTs. I need to rewrite them without lambda as we need to support JDK7 in branch-2.

          Show
          Apache9 Duo Zhang added a comment - can you upload branch-2.7 patch also..? Sure. Will upload later. I think the problem is the UTs. I need to rewrite them without lambda as we need to support JDK7 in branch-2.
          Hide
          Apache9 Duo Zhang added a comment -

          Rebase, and remove the CommonConfigurationKeys.HADOOP_KERBEROS_MIN_SECONDS_BEFORE_RELOGIN setting in testcase.

          Show
          Apache9 Duo Zhang added a comment - Rebase, and remove the CommonConfigurationKeys.HADOOP_KERBEROS_MIN_SECONDS_BEFORE_RELOGIN setting in testcase.
          Hide
          Apache9 Duo Zhang added a comment -

          Brahma Reddy Battula I tried and found that the patch can be cherry-picked to branch-2 cleanly. The problem is the two new UTs. After I rewrite the lambda part, they still can not pass. The problem is that we use a different kerberos implementation on branch-2. We use apache-kerby on trunk and apache-ds on branch-2. The KDC implementation of apache-ds will not reject the non-tgt request(maybe a bug).

          So I suggest you just cherry-pick the patch to your branch and remove the UTs to solve the problem quickly.

          Thanks.

          Show
          Apache9 Duo Zhang added a comment - Brahma Reddy Battula I tried and found that the patch can be cherry-picked to branch-2 cleanly. The problem is the two new UTs. After I rewrite the lambda part, they still can not pass. The problem is that we use a different kerberos implementation on branch-2. We use apache-kerby on trunk and apache-ds on branch-2. The KDC implementation of apache-ds will not reject the non-tgt request(maybe a bug). So I suggest you just cherry-pick the patch to your branch and remove the UTs to solve the problem quickly. Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 12m 31s trunk passed
          +1 compile 12m 46s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 1m 26s trunk passed
          +1 javadoc 0m 43s trunk passed
          +1 mvninstall 0m 38s the patch passed
          +1 compile 14m 21s the patch passed
          +1 javac 14m 21s the patch passed
          -0 checkstyle 0m 33s hadoop-common-project/hadoop-common: The patch generated 3 new + 94 unchanged - 2 fixed = 97 total (was 96)
          +1 mvnsite 1m 8s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 1m 54s the patch passed
          +1 javadoc 0m 52s the patch passed
          -1 unit 9m 13s hadoop-common in the patch failed.
          +1 asflicense 0m 31s The patch does not generate ASF License warnings.
          60m 36s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13433
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848011/HADOOP-13433-v5.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 29e10025f3d3 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e224c96
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11454/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11454/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11454/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11454/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 12m 31s trunk passed +1 compile 12m 46s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 26s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 38s the patch passed +1 compile 14m 21s the patch passed +1 javac 14m 21s the patch passed -0 checkstyle 0m 33s hadoop-common-project/hadoop-common: The patch generated 3 new + 94 unchanged - 2 fixed = 97 total (was 96) +1 mvnsite 1m 8s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 1m 54s the patch passed +1 javadoc 0m 52s the patch passed -1 unit 9m 13s hadoop-common in the patch failed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 60m 36s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13433 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848011/HADOOP-13433-v5.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 29e10025f3d3 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e224c96 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11454/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11454/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11454/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11454/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stack stack added a comment -

          So, patch is only suited to branch-3 Duo Zhang? Worth a version w/ UTs that will work for branch-2? What you think of the failed UT above? Is it related? Thanks.

          Show
          stack stack added a comment - So, patch is only suited to branch-3 Duo Zhang ? Worth a version w/ UTs that will work for branch-2? What you think of the failed UT above? Is it related? Thanks.
          Hide
          Apache9 Duo Zhang added a comment -

          stack The problem is the minikdc... We use apache-ds on branch-2 and it will not reject the wrong ticket... And I think minikdc is a seprated module, maybe we could backport the apache-kerby changes to branch-2 first?

          Thanks.

          Show
          Apache9 Duo Zhang added a comment - stack The problem is the minikdc... We use apache-ds on branch-2 and it will not reject the wrong ticket... And I think minikdc is a seprated module, maybe we could backport the apache-kerby changes to branch-2 first? Thanks.
          Hide
          Apache9 Duo Zhang added a comment -

          It is HADOOP-12911.

          Show
          Apache9 Duo Zhang added a comment - It is HADOOP-12911 .
          Hide
          stevel@apache.org Steve Loughran added a comment -

          It's only the tests which can't go into branch-2, correct? If so, don't worry about it: put the fix in the production code, and rely on trunk to host the tests

          Show
          stevel@apache.org Steve Loughran added a comment - It's only the tests which can't go into branch-2, correct? If so, don't worry about it: put the fix in the production code, and rely on trunk to host the tests
          Hide
          Apache9 Duo Zhang added a comment -

          Steve Loughran Thanks!

          Yes it is only the tests. I've just upload the patch for branch-2 which only contains fix for UGI.

          Show
          Apache9 Duo Zhang added a comment - Steve Loughran Thanks! Yes it is only the tests. I've just upload the patch for branch-2 which only contains fix for UGI.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 35s branch-2 passed
          +1 compile 5m 35s branch-2 passed with JDK v1.8.0_121
          +1 compile 6m 40s branch-2 passed with JDK v1.7.0_121
          +1 checkstyle 0m 26s branch-2 passed
          +1 mvnsite 0m 59s branch-2 passed
          +1 mvneclipse 0m 15s branch-2 passed
          +1 findbugs 1m 45s branch-2 passed
          +1 javadoc 0m 47s branch-2 passed with JDK v1.8.0_121
          +1 javadoc 0m 58s branch-2 passed with JDK v1.7.0_121
          +1 mvninstall 0m 40s the patch passed
          +1 compile 5m 42s the patch passed with JDK v1.8.0_121
          +1 javac 5m 42s the patch passed
          +1 compile 6m 28s the patch passed with JDK v1.7.0_121
          +1 javac 6m 28s the patch passed
          -0 checkstyle 0m 25s hadoop-common-project/hadoop-common: The patch generated 1 new + 94 unchanged - 2 fixed = 95 total (was 96)
          +1 mvnsite 0m 56s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 53s the patch passed
          +1 javadoc 0m 45s the patch passed with JDK v1.8.0_121
          +1 javadoc 0m 57s the patch passed with JDK v1.7.0_121
          +1 unit 8m 29s hadoop-common in the patch passed with JDK v1.7.0_121.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          61m 5s



          Reason Tests
          JDK v1.8.0_121 Failed junit tests hadoop.ipc.TestRPCWaitForProxy



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HADOOP-13433
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849290/HADOOP-13433-branch-2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 85ea8d7b3fa9 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 1af0a4d
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11506/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11506/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11506/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 35s branch-2 passed +1 compile 5m 35s branch-2 passed with JDK v1.8.0_121 +1 compile 6m 40s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 26s branch-2 passed +1 mvnsite 0m 59s branch-2 passed +1 mvneclipse 0m 15s branch-2 passed +1 findbugs 1m 45s branch-2 passed +1 javadoc 0m 47s branch-2 passed with JDK v1.8.0_121 +1 javadoc 0m 58s branch-2 passed with JDK v1.7.0_121 +1 mvninstall 0m 40s the patch passed +1 compile 5m 42s the patch passed with JDK v1.8.0_121 +1 javac 5m 42s the patch passed +1 compile 6m 28s the patch passed with JDK v1.7.0_121 +1 javac 6m 28s the patch passed -0 checkstyle 0m 25s hadoop-common-project/hadoop-common: The patch generated 1 new + 94 unchanged - 2 fixed = 95 total (was 96) +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 45s the patch passed with JDK v1.8.0_121 +1 javadoc 0m 57s the patch passed with JDK v1.7.0_121 +1 unit 8m 29s hadoop-common in the patch passed with JDK v1.7.0_121. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 61m 5s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.ipc.TestRPCWaitForProxy Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HADOOP-13433 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849290/HADOOP-13433-branch-2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 85ea8d7b3fa9 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 1af0a4d Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11506/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11506/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11506/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +1 for the branch-2 patch. As discussed, we can't backport the tests to branch-2

          Show
          stevel@apache.org Steve Loughran added a comment - +1 for the branch-2 patch. As discussed, we can't backport the tests to branch-2
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Committed to branch-2 & trunk.

          thanks!

          Show
          stevel@apache.org Steve Loughran added a comment - Committed to branch-2 & trunk. thanks!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11173 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11173/)
          HADOOP-13433 Race in UGI.reloginFromKeytab. Contributed by Duo Zhang. (stevel: rev 7fc3e68a876132563aa2321519fc6941e37b2cae)

          • (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestFixKerberosTicketOrder.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestRaceWhenRelogin.java
          • (edit) hadoop-common-project/hadoop-common/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11173 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11173/ ) HADOOP-13433 Race in UGI.reloginFromKeytab. Contributed by Duo Zhang. (stevel: rev 7fc3e68a876132563aa2321519fc6941e37b2cae) (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestFixKerberosTicketOrder.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestRaceWhenRelogin.java (edit) hadoop-common-project/hadoop-common/pom.xml
          Hide
          andrew.wang Andrew Wang added a comment -

          Please remember to set a 3.0.0-X version when committing to trunk, thanks!

          Show
          andrew.wang Andrew Wang added a comment - Please remember to set a 3.0.0-X version when committing to trunk, thanks!
          Hide
          stack stack added a comment -

          Thanks Steve Loughran for taking care of the commit.

          Show
          stack stack added a comment - Thanks Steve Loughran for taking care of the commit.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I strive to to keep the hbase team happy, though when it goes near UGI my enthusiasm is tempered by Fear of The Darkness

          Show
          stevel@apache.org Steve Loughran added a comment - I strive to to keep the hbase team happy, though when it goes near UGI my enthusiasm is tempered by Fear of The Darkness
          Hide
          yoderme Mike Yoder added a comment -

          This is a fascinating issue. Nice work catching and fixing it. Has anyone reported this to Oracle?

          Show
          yoderme Mike Yoder added a comment - This is a fascinating issue. Nice work catching and fixing it. Has anyone reported this to Oracle?
          Hide
          Apache9 Duo Zhang added a comment -

          I'm not sure if this should be considered as a bug of JDK...

          Searching for the tgt rather than using the first ticket as tgt can solve our problem here, but in general, there are also other races, for example, if you login and logout on the same subject object at the same time. I think the solution will be 'do not use it in that way'...

          Thanks.

          Show
          Apache9 Duo Zhang added a comment - I'm not sure if this should be considered as a bug of JDK... Searching for the tgt rather than using the first ticket as tgt can solve our problem here, but in general, there are also other races, for example, if you login and logout on the same subject object at the same time. I think the solution will be 'do not use it in that way'... Thanks.
          Hide
          hexiaoqiao He Xiaoqiao added a comment -

          Duo Zhang, after i patched in my production environment, there still some re-login exception as follow. (The env version - hadoop: 2.7.1, java: 1.7.0_76) Any suggestion? thank you.

          WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in offerService
          java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for hdfs/hostname@REALM to namenode/ip
          :8020; Host Details : local host is: "hostname/datanode"; destination host is: "namenode":8020; 
                  at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
                  at org.apache.hadoop.ipc.Client.call(Client.java:1474)
                  at org.apache.hadoop.ipc.Client.call(Client.java:1407)
                  at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
                  at com.sun.proxy.$Proxy13.sendHeartbeat(Unknown Source)
                  at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:153)
                  at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:554)
                  at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:653)
                  at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:824)
                  at java.lang.Thread.run(Thread.java:745)
          Caused by: java.io.IOException: Couldn't setup connection for hdfs/hostname@REALM to namenode/ip:8020
                  at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:674)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at javax.security.auth.Subject.doAs(Subject.java:415)
                  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690)
                  at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:645)
                  at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:732)
                  at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370)
                  at org.apache.hadoop.ipc.Client.getConnection(Client.java:1523)
                  at org.apache.hadoop.ipc.Client.call(Client.java:1446)
                  ... 8 more
          Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
                  at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
                  at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
                  at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:555)
                  at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:370)
                  at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:724)
                  at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:720)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at javax.security.auth.Subject.doAs(Subject.java:415)
                  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690)
                  at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:719)
                  ... 11 more
          Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
                  at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
                  at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
                  at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
                  at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
                  at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
                  at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
                  at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
                  ... 20 more
          WARN org.apache.hadoop.security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
          WARN org.apache.hadoop.security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
          
          Show
          hexiaoqiao He Xiaoqiao added a comment - Duo Zhang , after i patched in my production environment, there still some re-login exception as follow. (The env version - hadoop: 2.7.1, java: 1.7.0_76) Any suggestion? thank you. WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in offerService java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for hdfs/hostname@REALM to namenode/ip :8020; Host Details : local host is: "hostname/datanode" ; destination host is: "namenode" :8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773) at org.apache.hadoop.ipc.Client.call(Client.java:1474) at org.apache.hadoop.ipc.Client.call(Client.java:1407) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy13.sendHeartbeat(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:153) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:554) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:653) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:824) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Couldn't setup connection for hdfs/hostname@REALM to namenode/ip:8020 at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:674) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:645) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:732) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1523) at org.apache.hadoop.ipc.Client.call(Client.java:1446) ... 8 more Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:555) at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:370) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:724) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:720) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:719) ... 11 more Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) ... 20 more WARN org.apache.hadoop.security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before. WARN org.apache.hadoop.security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Created HADOOP-14037 to cover quality of the stack trace. Won't affect cause, but will stop exception type downgrading

          Show
          stevel@apache.org Steve Loughran added a comment - Created HADOOP-14037 to cover quality of the stack trace. Won't affect cause, but will stop exception type downgrading
          Hide
          Apache9 Duo Zhang added a comment -

          He Xiaoqiao Your exception is not the same with the description of this issue. The problem here is 'BAD TGS SERVER NAME'.

          You can add this to your command line to see what is going on when you try to relogin or setup a connection.

          -Dsun.security.krb5.debug=true
          

          Thanks.

          Show
          Apache9 Duo Zhang added a comment - He Xiaoqiao Your exception is not the same with the description of this issue. The problem here is 'BAD TGS SERVER NAME'. You can add this to your command line to see what is going on when you try to relogin or setup a connection. -Dsun.security.krb5.debug=true Thanks.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Hi Duo Zhang,

          Thanks for the nice work here. I have a question:

          // if the first kerberos ticket is not TGT, then remove and destroy it since
          // the kerberos library of jdk always use the first kerberos ticket as TGT.
          // See HADOOP-13433 for more details.
          @VisibleForTesting
          void fixKerberosTicketOrder() {

          Would you please explain why the tickets can be safely removed/destroyed rather than reordered? are these tickets that deemed useless?

          Other questions about your comment

          I'm not sure if this should be considered as a bug of JDK...
          Searching for the tgt rather than using the first ticket as tgt can solve our problem here, but in general, there are also other races, for example, if you login and logout on the same subject object at the same time. I think the solution will be 'do not use it in that way'...

          • Wonder if there is any jdk issue has been filed to request to search for tgt rather than using the first ticket?
          • Can the race condition be avoided with synchronization? if not, how can one avoid the situation or 'do not use it in that way'?

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Hi Duo Zhang , Thanks for the nice work here. I have a question: // if the first kerberos ticket is not TGT, then remove and destroy it since // the kerberos library of jdk always use the first kerberos ticket as TGT. // See HADOOP-13433 for more details. @VisibleForTesting void fixKerberosTicketOrder() { Would you please explain why the tickets can be safely removed/destroyed rather than reordered? are these tickets that deemed useless? Other questions about your comment I'm not sure if this should be considered as a bug of JDK... Searching for the tgt rather than using the first ticket as tgt can solve our problem here, but in general, there are also other races, for example, if you login and logout on the same subject object at the same time. I think the solution will be 'do not use it in that way'... Wonder if there is any jdk issue has been filed to request to search for tgt rather than using the first ticket? Can the race condition be avoided with synchronization? if not, how can one avoid the situation or 'do not use it in that way'? Thanks.
          Hide
          Apache9 Duo Zhang added a comment -

          Would you please explain why the tickets can be safely removed/destroyed rather than reordered? are these tickets that deemed useless?

          The implementation will create a new service ticket if not exists when you can doAs, so remove it is OK.

          Wonder if there is any jdk issue has been filed to request to search for tgt rather than using the first ticket?

          I do not know... At least I haven't report it to oracle yet...

          Can the race condition be avoided with synchronization? if not, how can one avoid the situation or 'do not use it in that way'?

          The former one yes. But it is a bit tricky as doAs may still modify the subject. And usually doing IO under lock is considered as bad practice.

          And the latter one, as I said above, make the subject volatile and create a new subject and do a switch.everytime when relogin, and then destroy the old subject after a graceful period. But this is also a bit difficult as UGI is declared as LimitedPrivacy, and we do have a method that allows construct a UGI from a subject, so it is not easy to switch subject when relogin without breaking the old code...

          Thanks.

          Show
          Apache9 Duo Zhang added a comment - Would you please explain why the tickets can be safely removed/destroyed rather than reordered? are these tickets that deemed useless? The implementation will create a new service ticket if not exists when you can doAs, so remove it is OK. Wonder if there is any jdk issue has been filed to request to search for tgt rather than using the first ticket? I do not know... At least I haven't report it to oracle yet... Can the race condition be avoided with synchronization? if not, how can one avoid the situation or 'do not use it in that way'? The former one yes. But it is a bit tricky as doAs may still modify the subject. And usually doing IO under lock is considered as bad practice. And the latter one, as I said above, make the subject volatile and create a new subject and do a switch.everytime when relogin, and then destroy the old subject after a graceful period. But this is also a bit difficult as UGI is declared as LimitedPrivacy, and we do have a method that allows construct a UGI from a subject, so it is not easy to switch subject when relogin without breaking the old code... Thanks.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the nice work/commit here Duo Zhang and Steve Loughran.

          The issue looks pretty important, and since others have asked above, can we backport the branch-2 change further to branch-2.8 and branch-2.7? Thanks.

          Show
          xiaochen Xiao Chen added a comment - Thanks for the nice work/commit here Duo Zhang and Steve Loughran . The issue looks pretty important, and since others have asked above, can we backport the branch-2 change further to branch-2.8 and branch-2.7? Thanks.
          Hide
          yzhangal Yongjun Zhang added a comment -

          Thanks Duo Zhang.

          The implementation will create a new service ticket if not exists when you can doAs, so remove it is OK.

          Wonder if that imply some performance loss because of the need to contact service for a new ticket? Instead of removing the item, if we simply put the item back to the end of the list, will that work?

          I guess this is a pretty corner case, so it would not be a big concern.

          Thanks.

          Show
          yzhangal Yongjun Zhang added a comment - Thanks Duo Zhang . The implementation will create a new service ticket if not exists when you can doAs, so remove it is OK. Wonder if that imply some performance loss because of the need to contact service for a new ticket? Instead of removing the item, if we simply put the item back to the end of the list, will that work? I guess this is a pretty corner case, so it would not be a big concern. Thanks.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          it wouldn't cherry pick into branch-2.8, so I didn't

          Show
          stevel@apache.org Steve Loughran added a comment - it wouldn't cherry pick into branch-2.8, so I didn't
          Hide
          Apache9 Duo Zhang added a comment -

          Let me prepare patches for branch-2.8 and branch-2.7.

          I guess this is a pretty corner case, so it would not be a big concern.

          Yes, this rarely happens so I do not think it is a big deal. And the service tickets are generated with the old tgt, so theoretically they should be destroyed.

          Thanks.

          Show
          Apache9 Duo Zhang added a comment - Let me prepare patches for branch-2.8 and branch-2.7. I guess this is a pretty corner case, so it would not be a big concern. Yes, this rarely happens so I do not think it is a big deal. And the service tickets are generated with the old tgt, so theoretically they should be destroyed. Thanks.
          Hide
          Apache9 Duo Zhang added a comment -

          Reopen to attach patches for branch-2.8 and branch-2.7.

          Show
          Apache9 Duo Zhang added a comment - Reopen to attach patches for branch-2.8 and branch-2.7.
          Hide
          Apache9 Duo Zhang added a comment -

          Patches for branch-2.8 and branch-2.7. The patch for branch-2.7 could also be applied to branch-2.6.

          Show
          Apache9 Duo Zhang added a comment - Patches for branch-2.8 and branch-2.7. The patch for branch-2.7 could also be applied to branch-2.6.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the backport patches, Duo Zhang!

          Seems jenkins only kicked https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/ when the 2 attachments were uploaded together, for 2.7.
          Downloaded an reattached the branch-2.8 patch to trigger jenkins (don't know how to kick the job for that patch otherwise..)

          Show
          xiaochen Xiao Chen added a comment - Thanks for the backport patches, Duo Zhang ! Seems jenkins only kicked https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/ when the 2 attachments were uploaded together, for 2.7. Downloaded an reattached the branch-2.8 patch to trigger jenkins (don't know how to kick the job for that patch otherwise..)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 12m 31s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 25s branch-2.7 passed
          +1 compile 6m 14s branch-2.7 passed with JDK v1.8.0_121
          +1 compile 7m 5s branch-2.7 passed with JDK v1.7.0_121
          +1 checkstyle 0m 29s branch-2.7 passed
          +1 mvnsite 0m 57s branch-2.7 passed
          +1 mvneclipse 0m 19s branch-2.7 passed
          -1 findbugs 1m 58s hadoop-common-project/hadoop-common in branch-2.7 has 3 extant Findbugs warnings.
          +1 javadoc 0m 56s branch-2.7 passed with JDK v1.8.0_121
          +1 javadoc 1m 6s branch-2.7 passed with JDK v1.7.0_121
          +1 mvninstall 0m 52s the patch passed
          +1 compile 7m 45s the patch passed with JDK v1.8.0_121
          +1 javac 7m 45s the patch passed
          +1 compile 7m 49s the patch passed with JDK v1.7.0_121
          +1 javac 7m 49s the patch passed
          -0 checkstyle 0m 29s hadoop-common-project/hadoop-common: The patch generated 2 new + 80 unchanged - 4 fixed = 82 total (was 84)
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 18s the patch passed
          -1 whitespace 0m 0s The patch has 4749 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 2m 28s The patch 98 line(s) with tabs.
          +1 findbugs 2m 14s the patch passed
          +1 javadoc 0m 54s the patch passed with JDK v1.8.0_121
          +1 javadoc 1m 10s the patch passed with JDK v1.7.0_121
          -1 unit 24m 35s hadoop-common in the patch failed with JDK v1.7.0_121.
          -1 asflicense 1m 0s The patch generated 1 ASF License warnings.
          114m 44s



          Reason Tests
          JDK v1.8.0_121 Failed junit tests hadoop.util.bloom.TestBloomFilters
            hadoop.ha.TestZKFailoverControllerStress
            hadoop.ipc.TestDecayRpcScheduler
            hadoop.ipc.TestCallQueueManager
            hadoop.metrics2.impl.TestGangliaMetrics
          JDK v1.8.0_121 Timed out junit tests org.apache.hadoop.conf.TestConfiguration
          JDK v1.7.0_121 Failed junit tests hadoop.util.bloom.TestBloomFilters
            hadoop.ha.TestZKFailoverControllerStress
            hadoop.ipc.TestCallQueueManager
          JDK v1.7.0_121 Timed out junit tests org.apache.hadoop.conf.TestConfiguration



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Issue HADOOP-13433
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850749/HADOOP-13433-branch-2.7.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6af601a529a8 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / 086ee0a
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 31s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 25s branch-2.7 passed +1 compile 6m 14s branch-2.7 passed with JDK v1.8.0_121 +1 compile 7m 5s branch-2.7 passed with JDK v1.7.0_121 +1 checkstyle 0m 29s branch-2.7 passed +1 mvnsite 0m 57s branch-2.7 passed +1 mvneclipse 0m 19s branch-2.7 passed -1 findbugs 1m 58s hadoop-common-project/hadoop-common in branch-2.7 has 3 extant Findbugs warnings. +1 javadoc 0m 56s branch-2.7 passed with JDK v1.8.0_121 +1 javadoc 1m 6s branch-2.7 passed with JDK v1.7.0_121 +1 mvninstall 0m 52s the patch passed +1 compile 7m 45s the patch passed with JDK v1.8.0_121 +1 javac 7m 45s the patch passed +1 compile 7m 49s the patch passed with JDK v1.7.0_121 +1 javac 7m 49s the patch passed -0 checkstyle 0m 29s hadoop-common-project/hadoop-common: The patch generated 2 new + 80 unchanged - 4 fixed = 82 total (was 84) +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 18s the patch passed -1 whitespace 0m 0s The patch has 4749 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 2m 28s The patch 98 line(s) with tabs. +1 findbugs 2m 14s the patch passed +1 javadoc 0m 54s the patch passed with JDK v1.8.0_121 +1 javadoc 1m 10s the patch passed with JDK v1.7.0_121 -1 unit 24m 35s hadoop-common in the patch failed with JDK v1.7.0_121. -1 asflicense 1m 0s The patch generated 1 ASF License warnings. 114m 44s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.util.bloom.TestBloomFilters   hadoop.ha.TestZKFailoverControllerStress   hadoop.ipc.TestDecayRpcScheduler   hadoop.ipc.TestCallQueueManager   hadoop.metrics2.impl.TestGangliaMetrics JDK v1.8.0_121 Timed out junit tests org.apache.hadoop.conf.TestConfiguration JDK v1.7.0_121 Failed junit tests hadoop.util.bloom.TestBloomFilters   hadoop.ha.TestZKFailoverControllerStress   hadoop.ipc.TestCallQueueManager JDK v1.7.0_121 Timed out junit tests org.apache.hadoop.conf.TestConfiguration Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue HADOOP-13433 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850749/HADOOP-13433-branch-2.7.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6af601a529a8 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 086ee0a Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 23m 28s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 0s branch-2.8 passed
          +1 compile 7m 4s branch-2.8 passed with JDK v1.8.0_121
          +1 compile 7m 18s branch-2.8 passed with JDK v1.7.0_121
          +1 checkstyle 0m 22s branch-2.8 passed
          +1 mvnsite 0m 58s branch-2.8 passed
          +1 mvneclipse 0m 16s branch-2.8 passed
          +1 findbugs 1m 38s branch-2.8 passed
          +1 javadoc 0m 45s branch-2.8 passed with JDK v1.8.0_121
          +1 javadoc 0m 57s branch-2.8 passed with JDK v1.7.0_121
          +1 mvninstall 0m 40s the patch passed
          +1 compile 6m 6s the patch passed with JDK v1.8.0_121
          +1 javac 6m 5s the patch passed
          +1 compile 7m 0s the patch passed with JDK v1.7.0_121
          +1 javac 7m 0s the patch passed
          -0 checkstyle 0m 21s hadoop-common-project/hadoop-common: The patch generated 1 new + 103 unchanged - 2 fixed = 104 total (was 105)
          +1 mvnsite 0m 57s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 49s the patch passed
          +1 javadoc 0m 44s the patch passed with JDK v1.8.0_121
          +1 javadoc 0m 55s the patch passed with JDK v1.7.0_121
          +1 unit 7m 58s hadoop-common in the patch passed with JDK v1.7.0_121.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          86m 45s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:5af2af1
          JIRA Issue HADOOP-13433
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850777/HADOOP-13433-branch-2.8.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 92138dfb0e0e 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.8 / 4e423ed
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11568/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11568/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11568/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 23m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 0s branch-2.8 passed +1 compile 7m 4s branch-2.8 passed with JDK v1.8.0_121 +1 compile 7m 18s branch-2.8 passed with JDK v1.7.0_121 +1 checkstyle 0m 22s branch-2.8 passed +1 mvnsite 0m 58s branch-2.8 passed +1 mvneclipse 0m 16s branch-2.8 passed +1 findbugs 1m 38s branch-2.8 passed +1 javadoc 0m 45s branch-2.8 passed with JDK v1.8.0_121 +1 javadoc 0m 57s branch-2.8 passed with JDK v1.7.0_121 +1 mvninstall 0m 40s the patch passed +1 compile 6m 6s the patch passed with JDK v1.8.0_121 +1 javac 6m 5s the patch passed +1 compile 7m 0s the patch passed with JDK v1.7.0_121 +1 javac 7m 0s the patch passed -0 checkstyle 0m 21s hadoop-common-project/hadoop-common: The patch generated 1 new + 103 unchanged - 2 fixed = 104 total (was 105) +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 44s the patch passed with JDK v1.8.0_121 +1 javadoc 0m 55s the patch passed with JDK v1.7.0_121 +1 unit 7m 58s hadoop-common in the patch passed with JDK v1.7.0_121. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 86m 45s Subsystem Report/Notes Docker Image:yetus/hadoop:5af2af1 JIRA Issue HADOOP-13433 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850777/HADOOP-13433-branch-2.8.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 92138dfb0e0e 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / 4e423ed Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11568/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11568/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11568/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          First of the checkstyles looks legit; the line ending one I'm not worried about

          ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java:77:import com.google.common.annotations.VisibleForTesting;:1: Duplicate import to line 23 - com.google.common.annotations.VisibleForTesting.
          
          Show
          stevel@apache.org Steve Loughran added a comment - First of the checkstyles looks legit; the line ending one I'm not worried about ./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java:77: import com.google.common.annotations.VisibleForTesting;:1: Duplicate import to line 23 - com.google.common.annotations.VisibleForTesting.
          Hide
          Apache9 Duo Zhang added a comment -

          The line ending issues are not introduced in the patch. They are already there...

          Let me remove the duplicated imports.

          Show
          Apache9 Duo Zhang added a comment - The line ending issues are not introduced in the patch. They are already there... Let me remove the duplicated imports.
          Hide
          Apache9 Duo Zhang added a comment -

          Remove duplicate imports.

          Show
          Apache9 Duo Zhang added a comment - Remove duplicate imports.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 11m 47s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 48s branch-2.7 passed
          +1 compile 5m 16s branch-2.7 passed with JDK v1.8.0_121
          +1 compile 6m 27s branch-2.7 passed with JDK v1.7.0_121
          +1 checkstyle 0m 25s branch-2.7 passed
          +1 mvnsite 0m 50s branch-2.7 passed
          +1 mvneclipse 0m 16s branch-2.7 passed
          -1 findbugs 1m 47s hadoop-common-project/hadoop-common in branch-2.7 has 3 extant Findbugs warnings.
          +1 javadoc 0m 47s branch-2.7 passed with JDK v1.8.0_121
          +1 javadoc 0m 56s branch-2.7 passed with JDK v1.7.0_121
          +1 mvninstall 0m 41s the patch passed
          +1 compile 6m 8s the patch passed with JDK v1.8.0_121
          +1 javac 6m 8s the patch passed
          +1 compile 6m 32s the patch passed with JDK v1.7.0_121
          +1 javac 6m 32s the patch passed
          -0 checkstyle 0m 23s hadoop-common-project/hadoop-common: The patch generated 1 new + 80 unchanged - 4 fixed = 81 total (was 84)
          +1 mvnsite 0m 45s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          -1 whitespace 0m 1s The patch has 5396 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 2m 20s The patch 98 line(s) with tabs.
          +1 findbugs 1m 45s the patch passed
          +1 javadoc 0m 40s the patch passed with JDK v1.8.0_121
          +1 javadoc 0m 52s the patch passed with JDK v1.7.0_121
          -1 unit 20m 29s hadoop-common in the patch failed with JDK v1.7.0_121.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          98m 36s



          Reason Tests
          JDK v1.8.0_121 Failed junit tests hadoop.util.bloom.TestBloomFilters
          JDK v1.8.0_121 Timed out junit tests org.apache.hadoop.conf.TestConfiguration
          JDK v1.7.0_121 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.util.bloom.TestBloomFilters
            hadoop.ipc.TestIPC



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Issue HADOOP-13433
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850811/HADOOP-13433-branch-2.7-v1.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 103dd8f94018 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / 086ee0a
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11570/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11570/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11570/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11570/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11570/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11570/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11570/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 47s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 48s branch-2.7 passed +1 compile 5m 16s branch-2.7 passed with JDK v1.8.0_121 +1 compile 6m 27s branch-2.7 passed with JDK v1.7.0_121 +1 checkstyle 0m 25s branch-2.7 passed +1 mvnsite 0m 50s branch-2.7 passed +1 mvneclipse 0m 16s branch-2.7 passed -1 findbugs 1m 47s hadoop-common-project/hadoop-common in branch-2.7 has 3 extant Findbugs warnings. +1 javadoc 0m 47s branch-2.7 passed with JDK v1.8.0_121 +1 javadoc 0m 56s branch-2.7 passed with JDK v1.7.0_121 +1 mvninstall 0m 41s the patch passed +1 compile 6m 8s the patch passed with JDK v1.8.0_121 +1 javac 6m 8s the patch passed +1 compile 6m 32s the patch passed with JDK v1.7.0_121 +1 javac 6m 32s the patch passed -0 checkstyle 0m 23s hadoop-common-project/hadoop-common: The patch generated 1 new + 80 unchanged - 4 fixed = 81 total (was 84) +1 mvnsite 0m 45s the patch passed +1 mvneclipse 0m 14s the patch passed -1 whitespace 0m 1s The patch has 5396 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 2m 20s The patch 98 line(s) with tabs. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 40s the patch passed with JDK v1.8.0_121 +1 javadoc 0m 52s the patch passed with JDK v1.7.0_121 -1 unit 20m 29s hadoop-common in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 98m 36s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.util.bloom.TestBloomFilters JDK v1.8.0_121 Timed out junit tests org.apache.hadoop.conf.TestConfiguration JDK v1.7.0_121 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.util.bloom.TestBloomFilters   hadoop.ipc.TestIPC Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue HADOOP-13433 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850811/HADOOP-13433-branch-2.7-v1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 103dd8f94018 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 086ee0a Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11570/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11570/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11570/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11570/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11570/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11570/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11570/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          Apache9 Duo Zhang added a comment -

          Any other concerns on the patches for branch-2.8 and branch-2.7? Xiao Chen Steve Loughran.

          Thanks.

          Show
          Apache9 Duo Zhang added a comment - Any other concerns on the patches for branch-2.8 and branch-2.7? Xiao Chen Steve Loughran . Thanks.
          Hide
          xiaochen Xiao Chen added a comment -

          Sorry for the delay Duo Zhang.

          The checkstyle look related for both patches, inside fixKerberosTicketOrder:

          /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java:1067: LOG.warn("Warning, no kerberos ticket found while attempting to renew ticket");: Line is longer than 80 characters (found 83).

          Whitespace and test failures on branch-2.7 look unrelated, so +1 to both pending the checkstyle fix...
          Thank you.

          Show
          xiaochen Xiao Chen added a comment - Sorry for the delay Duo Zhang . The checkstyle look related for both patches, inside fixKerberosTicketOrder : /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java:1067: LOG.warn("Warning, no kerberos ticket found while attempting to renew ticket");: Line is longer than 80 characters (found 83). Whitespace and test failures on branch-2.7 look unrelated, so +1 to both pending the checkstyle fix... Thank you.
          Hide
          Apache9 Duo Zhang added a comment -

          Fix the line length violation.

          Show
          Apache9 Duo Zhang added a comment - Fix the line length violation.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 30s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 21s branch-2.7 passed
          +1 compile 5m 43s branch-2.7 passed with JDK v1.8.0_121
          +1 compile 6m 38s branch-2.7 passed with JDK v1.7.0_121
          +1 checkstyle 0m 25s branch-2.7 passed
          +1 mvnsite 0m 51s branch-2.7 passed
          +1 mvneclipse 0m 18s branch-2.7 passed
          -1 findbugs 1m 45s hadoop-common-project/hadoop-common in branch-2.7 has 3 extant Findbugs warnings.
          +1 javadoc 0m 48s branch-2.7 passed with JDK v1.8.0_121
          +1 javadoc 0m 54s branch-2.7 passed with JDK v1.7.0_121
          +1 mvninstall 0m 41s the patch passed
          +1 compile 6m 9s the patch passed with JDK v1.8.0_121
          +1 javac 6m 9s the patch passed
          +1 compile 6m 15s the patch passed with JDK v1.7.0_121
          +1 javac 6m 15s the patch passed
          +1 checkstyle 0m 22s hadoop-common-project/hadoop-common: The patch generated 0 new + 80 unchanged - 4 fixed = 80 total (was 84)
          +1 mvnsite 0m 46s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          -1 whitespace 0m 0s The patch has 5748 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 2m 23s The patch 98 line(s) with tabs.
          +1 findbugs 1m 45s the patch passed
          +1 javadoc 0m 43s the patch passed with JDK v1.8.0_121
          +1 javadoc 1m 45s the patch passed with JDK v1.7.0_121
          -1 unit 16m 5s hadoop-common in the patch failed with JDK v1.7.0_121.
          +1 asflicense 0m 35s The patch does not generate ASF License warnings.
          81m 24s



          Reason Tests
          JDK v1.8.0_121 Failed junit tests hadoop.fs.TestS3_LocalFileContextURI
            hadoop.util.bloom.TestBloomFilters
            hadoop.io.compress.TestCodecPool
            hadoop.fs.TestLocal_S3FileContextURI
            hadoop.ipc.TestIPC
          JDK v1.8.0_121 Timed out junit tests org.apache.hadoop.conf.TestConfiguration
          JDK v1.7.0_121 Failed junit tests hadoop.security.ssl.TestSSLFactory
            hadoop.fs.TestS3_LocalFileContextURI
            hadoop.util.bloom.TestBloomFilters
            hadoop.fs.TestLocal_S3FileContextURI
            hadoop.ipc.TestIPC
          JDK v1.7.0_121 Timed out junit tests org.apache.hadoop.conf.TestConfiguration



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Issue HADOOP-13433
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12851608/HADOOP-13433-branch-2.7-v2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 98a82c31d0ae 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / bc3e188
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11597/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11597/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11597/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11597/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11597/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11597/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 30s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 21s branch-2.7 passed +1 compile 5m 43s branch-2.7 passed with JDK v1.8.0_121 +1 compile 6m 38s branch-2.7 passed with JDK v1.7.0_121 +1 checkstyle 0m 25s branch-2.7 passed +1 mvnsite 0m 51s branch-2.7 passed +1 mvneclipse 0m 18s branch-2.7 passed -1 findbugs 1m 45s hadoop-common-project/hadoop-common in branch-2.7 has 3 extant Findbugs warnings. +1 javadoc 0m 48s branch-2.7 passed with JDK v1.8.0_121 +1 javadoc 0m 54s branch-2.7 passed with JDK v1.7.0_121 +1 mvninstall 0m 41s the patch passed +1 compile 6m 9s the patch passed with JDK v1.8.0_121 +1 javac 6m 9s the patch passed +1 compile 6m 15s the patch passed with JDK v1.7.0_121 +1 javac 6m 15s the patch passed +1 checkstyle 0m 22s hadoop-common-project/hadoop-common: The patch generated 0 new + 80 unchanged - 4 fixed = 80 total (was 84) +1 mvnsite 0m 46s the patch passed +1 mvneclipse 0m 14s the patch passed -1 whitespace 0m 0s The patch has 5748 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 2m 23s The patch 98 line(s) with tabs. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 43s the patch passed with JDK v1.8.0_121 +1 javadoc 1m 45s the patch passed with JDK v1.7.0_121 -1 unit 16m 5s hadoop-common in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 81m 24s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.fs.TestS3_LocalFileContextURI   hadoop.util.bloom.TestBloomFilters   hadoop.io.compress.TestCodecPool   hadoop.fs.TestLocal_S3FileContextURI   hadoop.ipc.TestIPC JDK v1.8.0_121 Timed out junit tests org.apache.hadoop.conf.TestConfiguration JDK v1.7.0_121 Failed junit tests hadoop.security.ssl.TestSSLFactory   hadoop.fs.TestS3_LocalFileContextURI   hadoop.util.bloom.TestBloomFilters   hadoop.fs.TestLocal_S3FileContextURI   hadoop.ipc.TestIPC JDK v1.7.0_121 Timed out junit tests org.apache.hadoop.conf.TestConfiguration Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue HADOOP-13433 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12851608/HADOOP-13433-branch-2.7-v2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 98a82c31d0ae 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / bc3e188 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11597/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11597/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11597/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11597/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11597/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11597/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for revving Duo Zhang, +1 to both, committing them.

          Show
          xiaochen Xiao Chen added a comment - Thanks for revving Duo Zhang , +1 to both, committing them.
          Hide
          xiaochen Xiao Chen added a comment -

          Committed to branch-2.8, branch-2.7 and branch-2.6. Thanks again, Duo!

          Show
          xiaochen Xiao Chen added a comment - Committed to branch-2.8, branch-2.7 and branch-2.6. Thanks again, Duo!
          Hide
          Apache9 Duo Zhang added a comment -

          My pleasure.

          Show
          Apache9 Duo Zhang added a comment - My pleasure.
          Hide
          daryn Daryn Sharp added a comment -

          Just saw this jira due to internal conflict. We discovered the exact same issue a few years ago. This jira's patch works around the side-effect of the root cause: the entire relogin process is not atomic with regard to gssapi or spnego authentication. The jdk issue is technically a bug, but it's only triggered by hadoop's unsafe subject manipulation.

          I was already intending to release our internal fix within a week. It should negate the need for this patch but this can still be a safety net.

          Show
          daryn Daryn Sharp added a comment - Just saw this jira due to internal conflict. We discovered the exact same issue a few years ago. This jira's patch works around the side-effect of the root cause: the entire relogin process is not atomic with regard to gssapi or spnego authentication. The jdk issue is technically a bug, but it's only triggered by hadoop's unsafe subject manipulation. I was already intending to release our internal fix within a week. It should negate the need for this patch but this can still be a safety net.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          The added test TestRaceWhenRelogin failed occasionally in a few precommit jobs. Would any one be interested in looking into this? I filed HADOOP-14078 to track it. Thanks!

          Show
          jojochuang Wei-Chiu Chuang added a comment - The added test TestRaceWhenRelogin failed occasionally in a few precommit jobs. Would any one be interested in looking into this? I filed HADOOP-14078 to track it. Thanks!
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Daryn Sharp would you mind to file a jira for your fix?
          Duo Zhang found an issue with the test case that seems to stem from a JDK bug. Not sure if that's what you observed. My fear is HADOOP-13433 might not be a complete solution to this bug.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Daryn Sharp would you mind to file a jira for your fix? Duo Zhang found an issue with the test case that seems to stem from a JDK bug. Not sure if that's what you observed. My fear is HADOOP-13433 might not be a complete solution to this bug.
          Hide
          zhz Zhe Zhang added a comment -

          Daryn Sharp Any chance you can upload the internal fix? I'll be very happy to help review. Thanks.

          Show
          zhz Zhe Zhang added a comment - Daryn Sharp Any chance you can upload the internal fix? I'll be very happy to help review. Thanks.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

            People

            • Assignee:
              Apache9 Duo Zhang
              Reporter:
              Apache9 Duo Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              32 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development