Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9276

Failed to Update HDFS Delegation Token for long running application in HA mode

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: fs, ha, security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The Scenario is as follows:
      1. NameNode HA is enabled.
      2. Kerberos is enabled.
      3. HDFS Delegation Token (not Keytab or TGT) is used to communicate with NameNode.
      4. We want to update the HDFS Delegation Token for long running applicatons. HDFS Client will generate private tokens for each NameNode. When we update the HDFS Delegation Token, these private tokens will not be updated, which will cause token expired.

      This bug can be reproduced by the following program:

      import java.security.PrivilegedExceptionAction
      import org.apache.hadoop.conf.Configuration
      import org.apache.hadoop.fs.{FileSystem, Path}
      import org.apache.hadoop.security.UserGroupInformation
      
      object HadoopKerberosTest {
      
        def main(args: Array[String]): Unit = {
          val keytab = "/path/to/keytab/xxx.keytab"
          val principal = "xxx@ABC.COM"
      
          val creds1 = new org.apache.hadoop.security.Credentials()
          val ugi1 = UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
          ugi1.doAs(new PrivilegedExceptionAction[Void] {
            // Get a copy of the credentials
            override def run(): Void = {
              val fs = FileSystem.get(new Configuration())
              fs.addDelegationTokens("test", creds1)
              null
            }
          })
      
          val ugi = UserGroupInformation.createRemoteUser("test")
          ugi.addCredentials(creds1)
          ugi.doAs(new PrivilegedExceptionAction[Void] {
            // Get a copy of the credentials
            override def run(): Void = {
              var i = 0
              while (true) {
                val creds1 = new org.apache.hadoop.security.Credentials()
                val ugi1 = UserGroupInformation.loginUserFromKeytabAndReturnUGI(principal, keytab)
                ugi1.doAs(new PrivilegedExceptionAction[Void] {
                  // Get a copy of the credentials
                  override def run(): Void = {
                    val fs = FileSystem.get(new Configuration())
                    fs.addDelegationTokens("test", creds1)
                    null
                  }
                })
                UserGroupInformation.getCurrentUser.addCredentials(creds1)
      
                val fs = FileSystem.get( new Configuration())
                i += 1
                println()
                println(i)
                println(fs.listFiles(new Path("/user"), false))
                Thread.sleep(60 * 1000)
              }
              null
            }
          })
        }
      }
      

      To reproduce the bug, please set the following configuration to Name Node:

      dfs.namenode.delegation.token.max-lifetime = 10min
      dfs.namenode.delegation.key.update-interval = 3min
      dfs.namenode.delegation.token.renew-interval = 3min
      

      The bug will occure after 3 minutes.

      The stacktrace is:

      Exception in thread "main" org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 330156 for test) is expired
      	at org.apache.hadoop.ipc.Client.call(Client.java:1347)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1300)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
      	at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:651)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
      	at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
      	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1679)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1106)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
      	at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:747)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$15.<init>(DistributedFileSystem.java:726)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:717)
      	at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1780)
      	at org.apache.hadoop.fs.FileSystem$5.<init>(FileSystem.java:1842)
      	at org.apache.hadoop.fs.FileSystem.listFiles(FileSystem.java:1839)
      	at HadoopKerberosTest6$$anon$2.run(HadoopKerberosTest6.scala:55)
      	at HadoopKerberosTest6$$anon$2.run(HadoopKerberosTest6.scala:32)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
      	at HadoopKerberosTest6$.main(HadoopKerberosTest6.scala:32)
      	at HadoopKerberosTest6.main(HadoopKerberosTest6.scala)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
      
      1. debug1.PNG
        448 kB
        Liangliang Gu
      2. debug2.PNG
        163 kB
        Liangliang Gu
      3. HDFS-9276.01.patch
        3 kB
        Liangliang Gu
      4. HDFS-9276.02.patch
        3 kB
        Liangliang Gu
      5. HDFS-9276.03.patch
        3 kB
        Liangliang Gu
      6. HDFS-9276.04.patch
        6 kB
        Liangliang Gu
      7. HDFS-9276.05.patch
        5 kB
        Liangliang Gu
      8. HDFS-9276.06.patch
        8 kB
        Liangliang Gu
      9. HDFS-9276.07.patch
        8 kB
        Liangliang Gu
      10. HDFS-9276.08.patch
        7 kB
        Liangliang Gu
      11. HDFS-9276.09.patch
        7 kB
        Liangliang Gu
      12. HDFS-9276.10.patch
        6 kB
        Liangliang Gu
      13. HDFS-9276.11.patch
        6 kB
        Liangliang Gu
      14. HDFS-9276.12.patch
        6 kB
        Liangliang Gu
      15. HDFS-9276.13.patch
        6 kB
        Liangliang Gu
      16. HDFS-9276.14.patch
        7 kB
        John Zhuge
      17. HDFS-9276.15.patch
        6 kB
        John Zhuge
      18. HDFS-9276.16.patch
        6 kB
        John Zhuge
      19. HDFS-9276.17.patch
        6 kB
        John Zhuge
      20. HDFS-9276.18.patch
        6 kB
        John Zhuge
      21. HDFS-9276.19.patch
        7 kB
        John Zhuge
      22. HDFS-9276.20.patch
        7 kB
        John Zhuge
      23. HDFSReadLoop.scala
        0.6 kB
        John Zhuge

        Issue Links

          Activity

          Hide
          marsishandsome Liangliang Gu added a comment -

          org.apache.hadoop.hdfs.HAUtil#cloneDelegationTokenForLogicalUri will copy Token for NameNode HA to Token for NameNode 1 and Token for NameNode 2.
          Token for NameNode 1 and Token for NameNode 2 are private Tokens, while Token for NameNode HA is not.

          Token for NameNode 1 will be selected when communicate with NameNode 1, while Token for NameNode 2 will be selected when communicate with NameNode 2.

          The debug info is catched at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSelector#selectToken.

          In long running applications, the tokens will be expired, so we must update the tokens.
          If Token for NameNode HA is updated by calling UserGroupInformation.getCurrentUser.addCredentials, Token for NameNode 1 and Token for NameNode 2 will not be updated automatically, which will cause token expired error.

          Show
          marsishandsome Liangliang Gu added a comment - org.apache.hadoop.hdfs.HAUtil#cloneDelegationTokenForLogicalUri will copy Token for NameNode HA to Token for NameNode 1 and Token for NameNode 2 . Token for NameNode 1 and Token for NameNode 2 are private Tokens, while Token for NameNode HA is not. Token for NameNode 1 will be selected when communicate with NameNode 1, while Token for NameNode 2 will be selected when communicate with NameNode 2. The debug info is catched at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSelector#selectToken . In long running applications, the tokens will be expired, so we must update the tokens. If Token for NameNode HA is updated by calling UserGroupInformation.getCurrentUser.addCredentials , Token for NameNode 1 and Token for NameNode 2 will not be updated automatically, which will cause token expired error.
          Hide
          marsishandsome Liangliang Gu added a comment -

          Show
          marsishandsome Liangliang Gu added a comment -
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 22s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 9m 11s There were no new javac warning messages.
          +1 javadoc 12m 15s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 15s The applied patch generated 1 new checkstyle issues (total was 28, now 29).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 46s mvn install still works.
          +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
          -1 findbugs 2m 9s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          -1 common tests 8m 2s Tests failed in hadoop-common.
              55m 6s  



          Reason Tests
          FindBugs module:hadoop-common
          Failed unit tests hadoop.ipc.TestDecayRpcScheduler



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12768233/HDFS-9276.01.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 124a412
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13148/artifact/patchprocess/diffcheckstylehadoop-common.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13148/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13148/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13148/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13148/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 22s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 9m 11s There were no new javac warning messages. +1 javadoc 12m 15s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 15s The applied patch generated 1 new checkstyle issues (total was 28, now 29). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 46s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. -1 findbugs 2m 9s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. -1 common tests 8m 2s Tests failed in hadoop-common.     55m 6s   Reason Tests FindBugs module:hadoop-common Failed unit tests hadoop.ipc.TestDecayRpcScheduler Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768233/HDFS-9276.01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 124a412 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13148/artifact/patchprocess/diffcheckstylehadoop-common.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13148/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13148/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13148/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13148/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12768266/HDFS-9276.02.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 124a412
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13150/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768266/HDFS-9276.02.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 124a412 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13150/console This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          give the patch is all to hadoop common -filing a JIRA & patch there will run the hadoop-common build & test, which is a bit less brittle than the HDFS one

          Show
          stevel@apache.org Steve Loughran added a comment - give the patch is all to hadoop common -filing a JIRA & patch there will run the hadoop-common build & test, which is a bit less brittle than the HDFS one
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 1s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 8m 38s There were no new javac warning messages.
          +1 javadoc 11m 58s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 13s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 41s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          -1 findbugs 2m 23s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 common tests 8m 56s Tests passed in hadoop-common.
              54m 52s  



          Reason Tests
          FindBugs module:hadoop-common



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12768270/HDFS-9276.03.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 124a412
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13151/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13151/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13151/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13151/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 1s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 38s There were no new javac warning messages. +1 javadoc 11m 58s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 13s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 41s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. -1 findbugs 2m 23s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 common tests 8m 56s Tests passed in hadoop-common.     54m 52s   Reason Tests FindBugs module:hadoop-common Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768270/HDFS-9276.03.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 124a412 Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13151/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13151/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13151/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13151/console This message was automatically generated.
          Hide
          hitliuyi Yi Liu added a comment - - edited

          Liangliang Gu, Agree with Steve you need to move this to Hadoop common if the patch only contains change in common. Before this, I think you may have a mistake about how to use delegation token.

          Do you want to update the delegation token through FileSystem#addDelegationTokens? It will not get new delegation token again if old one exists in the credentials, also it may be more complicate than what you think.
          Actually I am curious about how you write your long running application. Is your application just an user client or a separate service? If your application is just one user client, I mean it's not a service which is accessed by many user clients, then you still need to user Kerberos instead of delegationToken, but if your application is a real service which serves user clients, then the delegation token is the right one. The delegation token is used in your service to access HDFS on behalf the user, usually your application service can renew the delegation token, the application service itself can't get a new delegation token for some specific user. If your application service runs longer than the maximum renewable date of user's delegationToken, one way is the user gets a new delegation token and your application service supports some mechanism to let user to update the delegation token and then refresh the token in that user's UGI's credential. Another way is to support proxy user privileges in your running application, refer to YARN-2704. Are you in the correct way?

          Show
          hitliuyi Yi Liu added a comment - - edited Liangliang Gu , Agree with Steve you need to move this to Hadoop common if the patch only contains change in common. Before this, I think you may have a mistake about how to use delegation token. Do you want to update the delegation token through FileSystem#addDelegationTokens ? It will not get new delegation token again if old one exists in the credentials, also it may be more complicate than what you think. Actually I am curious about how you write your long running application. Is your application just an user client or a separate service? If your application is just one user client, I mean it's not a service which is accessed by many user clients, then you still need to user Kerberos instead of delegationToken, but if your application is a real service which serves user clients, then the delegation token is the right one. The delegation token is used in your service to access HDFS on behalf the user, usually your application service can renew the delegation token, the application service itself can't get a new delegation token for some specific user. If your application service runs longer than the maximum renewable date of user's delegationToken, one way is the user gets a new delegation token and your application service supports some mechanism to let user to update the delegation token and then refresh the token in that user's UGI's credential. Another way is to support proxy user privileges in your running application, refer to YARN-2704 . Are you in the correct way?
          Hide
          marsishandsome Liangliang Gu added a comment -

          In my test code, a new UGI is created and the hdfs delegation token is got from the new UGI, so a new hdfs delegation token will be returned.
          You can reproduce the bug according to the test code I provided.

          I find this bug when running spark streaming in yarn-cluster mode using "--principal --keytab" argument (more then 7 days).
          This jira https://issues.apache.org/jira/browse/SPARK-5342 shows how spark get new hdfs delegation token and update to the current ugi.

          This jira https://issues.apache.org/jira/browse/SPARK-8688 fixes the bug when updating the token to the current ugi.
          But it only fixes the bug in the application master. This bug will also occure in the executor, when the executor updates the token.

          My patch wants to fixed the bug in hadoop, so spark does not need to do some workaround.

          Show
          marsishandsome Liangliang Gu added a comment - In my test code, a new UGI is created and the hdfs delegation token is got from the new UGI, so a new hdfs delegation token will be returned. You can reproduce the bug according to the test code I provided. I find this bug when running spark streaming in yarn-cluster mode using "--principal --keytab" argument (more then 7 days). This jira https://issues.apache.org/jira/browse/SPARK-5342 shows how spark get new hdfs delegation token and update to the current ugi. This jira https://issues.apache.org/jira/browse/SPARK-8688 fixes the bug when updating the token to the current ugi. But it only fixes the bug in the application master. This bug will also occure in the executor, when the executor updates the token. My patch wants to fixed the bug in hadoop, so spark does not need to do some workaround.
          Hide
          hitliuyi Yi Liu added a comment - - edited

          To reproduce the bug, please set the following configuration to Name Node:
          dfs.namenode.delegation.token.max-lifetime = 10min
          dfs.namenode.delegation.key.update-interval = 3min
          dfs.namenode.delegation.token.renew-interval = 3min
          The bug will occure after 3 minutes.

          Your test code can't say anything, the error msg of "token (HDFS_DELEGATION_TOKEN token 330156 for test) is expired" is because you set "dfs.namenode.delegation.token.renew-interval" to 3 min but you don't let test user to renew the token.

          I see what you want to do now, it's the same with the later case of what I commented above. Actually hadoop code is enough to let you do what you want to do. If a user client get a new delegation token, and your long running application can accept it, you can update the credentials of user's UGI on the server through UserGroupInformation#addCredentials, it will overwrite old tokens by default, of course you should make the service name of token is the same if you want to overwrite it.

          It's not a bug.

          Show
          hitliuyi Yi Liu added a comment - - edited To reproduce the bug, please set the following configuration to Name Node: dfs.namenode.delegation.token.max-lifetime = 10min dfs.namenode.delegation.key.update-interval = 3min dfs.namenode.delegation.token.renew-interval = 3min The bug will occure after 3 minutes. Your test code can't say anything, the error msg of "token (HDFS_DELEGATION_TOKEN token 330156 for test) is expired" is because you set "dfs.namenode.delegation.token.renew-interval" to 3 min but you don't let test user to renew the token. I see what you want to do now, it's the same with the later case of what I commented above. Actually hadoop code is enough to let you do what you want to do. If a user client get a new delegation token, and your long running application can accept it, you can update the credentials of user's UGI on the server through UserGroupInformation#addCredentials , it will overwrite old tokens by default, of course you should make the service name of token is the same if you want to overwrite it. It's not a bug.
          Hide
          marsishandsome Liangliang Gu added a comment -

          FileSystem#addDelegationTokens will return Token for NameNode HA.
          Is there any public and stable API to update Token for NameNode 1 and Token for NameNode2?

          Show
          marsishandsome Liangliang Gu added a comment - FileSystem#addDelegationTokens will return Token for NameNode HA . Is there any public and stable API to update Token for NameNode 1 and Token for NameNode2 ?
          Hide
          marsishandsome Liangliang Gu added a comment -

          ps: Token for NameNode 1 and Token for NameNode2 is generated by DFSClient and is PrivateToken.

          Show
          marsishandsome Liangliang Gu added a comment - ps: Token for NameNode 1 and Token for NameNode2 is generated by DFSClient and is PrivateToken.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          To collrelat with Liangliang Gu, Spark in long-lived service mode has a keytab for the AM, which then refreshes and re-issues delegation tokens which it hands out to worker containers. This is apparently not working on HA HDFS clusters.

          Show
          stevel@apache.org Steve Loughran added a comment - To collrelat with Liangliang Gu , Spark in long-lived service mode has a keytab for the AM, which then refreshes and re-issues delegation tokens which it hands out to worker containers. This is apparently not working on HA HDFS clusters.
          Hide
          hitliuyi Yi Liu added a comment -

          So it's the problem of service name of token for HA, there are three entries for the Token of NN HA with different service names: fs uri, NN1 uri, NN2 uri.
          The bug is only for HA mode, please don't use that test in the JIRA description, it has no relationship with HA.

          Also you need to add a real test to reproduce the issue which only occurs in HA in the patch.

          For the patch itself, we don't need Token#createPrivateToken, can construct it directly.
          Rename copyFrom to publicService, clonedToken to privateToken

          Show
          hitliuyi Yi Liu added a comment - So it's the problem of service name of token for HA, there are three entries for the Token of NN HA with different service names: fs uri, NN1 uri, NN2 uri. The bug is only for HA mode, please don't use that test in the JIRA description, it has no relationship with HA. Also you need to add a real test to reproduce the issue which only occurs in HA in the patch. For the patch itself, we don't need Token#createPrivateToken , can construct it directly. Rename copyFrom to publicService , clonedToken to privateToken
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 46s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 16s There were no new javac warning messages.
          +1 javadoc 10m 55s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 16s The applied patch generated 3 new checkstyle issues (total was 28, now 31).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 2m 0s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 common tests 7m 29s Tests passed in hadoop-common.
              50m 13s  



          Reason Tests
          FindBugs module:hadoop-common



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12768888/HDFS-9276.04.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 96677be
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13218/artifact/patchprocess/diffcheckstylehadoop-common.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13218/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13218/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13218/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13218/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 46s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 16s There were no new javac warning messages. +1 javadoc 10m 55s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 16s The applied patch generated 3 new checkstyle issues (total was 28, now 31). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 2m 0s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 common tests 7m 29s Tests passed in hadoop-common.     50m 13s   Reason Tests FindBugs module:hadoop-common Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768888/HDFS-9276.04.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 96677be checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13218/artifact/patchprocess/diffcheckstylehadoop-common.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13218/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13218/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13218/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13218/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 23m 25s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 12m 50s There were no new javac warning messages.
          +1 javadoc 18m 22s There were no new javadoc warning messages.
          +1 release audit 0m 44s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 54s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 2m 35s mvn install still works.
          +1 eclipse:eclipse 0m 58s The patch built with eclipse:eclipse.
          -1 findbugs 3m 31s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          -1 common tests 10m 55s Tests failed in hadoop-common.
              75m 18s  



          Reason Tests
          FindBugs module:hadoop-common
          Failed unit tests hadoop.fs.TestLocalFsFCStatistics
            hadoop.ipc.TestDecayRpcScheduler
            hadoop.ipc.TestRPCWaitForProxy



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12768947/HDFS-9276.05.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 96677be
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13221/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13221/artifact/patchprocess/testrun_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13221/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13221/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 23m 25s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 12m 50s There were no new javac warning messages. +1 javadoc 18m 22s There were no new javadoc warning messages. +1 release audit 0m 44s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 54s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 2m 35s mvn install still works. +1 eclipse:eclipse 0m 58s The patch built with eclipse:eclipse. -1 findbugs 3m 31s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. -1 common tests 10m 55s Tests failed in hadoop-common.     75m 18s   Reason Tests FindBugs module:hadoop-common Failed unit tests hadoop.fs.TestLocalFsFCStatistics   hadoop.ipc.TestDecayRpcScheduler   hadoop.ipc.TestRPCWaitForProxy Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768947/HDFS-9276.05.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 96677be Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/13221/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13221/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13221/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13221/console This message was automatically generated.
          Hide
          hitliuyi Yi Liu added a comment - - edited

          Now looks good overall.
          1.

          +
          +  /**
          +   * Create a private token for HA failover proxy.
          +   * @return the private token
          +   */
          +  public PrivateToken<T> createPrivateToken() {
          +    return new PrivateToken<T>(this);
          +  }
          

          Now, we can remove this.

          2.

          -    } else if (right == null || getClass() != right.getClass()) {
          +    } else if (right == null || !(right instanceof Token)) {
          

          Any reason to change this? It violates equals definition.

          3.
          We should have test for real case, maybe like following:

          • build a MiniDFSCluster with HA and security enabled.
          • Gets delegation token
          • Use the delegation token to access HDFS.
          • Invalid the delegation token in NN. Check the access failed.
          • Get a new delegation token and update through user's UGI.
          • Check the access again, should be successful.

          You can refer to existing tests about how to enable HA and security.

          Show
          hitliuyi Yi Liu added a comment - - edited Now looks good overall. 1. + + /** + * Create a private token for HA failover proxy. + * @ return the private token + */ + public PrivateToken<T> createPrivateToken() { + return new PrivateToken<T>( this ); + } Now, we can remove this. 2. - } else if (right == null || getClass() != right.getClass()) { + } else if (right == null || !(right instanceof Token)) { Any reason to change this? It violates equals definition. 3. We should have test for real case, maybe like following: build a MiniDFSCluster with HA and security enabled. Gets delegation token Use the delegation token to access HDFS. Invalid the delegation token in NN. Check the access failed. Get a new delegation token and update through user's UGI. Check the access again, should be successful. You can refer to existing tests about how to enable HA and security.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 6s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 14s trunk passed
          +1 compile 4m 43s trunk passed with JDK v1.8.0_60
          +1 compile 4m 33s trunk passed with JDK v1.7.0_79
          +1 checkstyle 1m 6s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          -1 findbugs 2m 3s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 2m 16s trunk passed with JDK v1.8.0_60
          +1 javadoc 3m 4s trunk passed with JDK v1.7.0_79
          +1 mvninstall 2m 20s the patch passed
          +1 compile 4m 41s the patch passed with JDK v1.8.0_60
          +1 javac 4m 41s the patch passed
          +1 compile 4m 29s the patch passed with JDK v1.7.0_79
          +1 javac 4m 29s the patch passed
          +1 checkstyle 1m 2s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 findbugs 2m 2s hadoop-common-project/hadoop-common introduced 1 new FindBugs issues.
          +1 javadoc 2m 11s the patch passed with JDK v1.8.0_60
          +1 javadoc 3m 6s the patch passed with JDK v1.7.0_79
          -1 unit 7m 27s hadoop-common in the patch failed with JDK v1.8.0_60.
          -1 unit 55m 48s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          -1 unit 19m 15s hadoop-common in the patch failed with JDK v1.7.0_79.
          -1 unit 57m 46s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 21s Patch generated 58 ASF License warnings.
          187m 55s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            org.apache.hadoop.security.token.Token$PrivateToken doesn't override Token.equals(Object) At Token.java:At Token.java:[line 1]
          JDK v1.7.0_79 Failed junit tests hadoop.ipc.TestRPC
            hadoop.ha.TestZKFailoverController
            hadoop.hdfs.server.datanode.TestFsDatasetCache
            hadoop.hdfs.TestLeaseRecovery2
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.namenode.TestNamenodeRetryCache
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
          JDK v1.7.0_79 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-29
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769535/HDFS-9276.07.patch
          JIRA Issue HDFS-9276
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux ac6e55b587f6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-67f42f1/precommit/personality/hadoop.sh
          git revision trunk / c416999
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13276/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 225MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13276/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 14s trunk passed +1 compile 4m 43s trunk passed with JDK v1.8.0_60 +1 compile 4m 33s trunk passed with JDK v1.7.0_79 +1 checkstyle 1m 6s trunk passed +1 mvneclipse 0m 27s trunk passed -1 findbugs 2m 3s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 2m 16s trunk passed with JDK v1.8.0_60 +1 javadoc 3m 4s trunk passed with JDK v1.7.0_79 +1 mvninstall 2m 20s the patch passed +1 compile 4m 41s the patch passed with JDK v1.8.0_60 +1 javac 4m 41s the patch passed +1 compile 4m 29s the patch passed with JDK v1.7.0_79 +1 javac 4m 29s the patch passed +1 checkstyle 1m 2s the patch passed +1 mvneclipse 0m 28s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 findbugs 2m 2s hadoop-common-project/hadoop-common introduced 1 new FindBugs issues. +1 javadoc 2m 11s the patch passed with JDK v1.8.0_60 +1 javadoc 3m 6s the patch passed with JDK v1.7.0_79 -1 unit 7m 27s hadoop-common in the patch failed with JDK v1.8.0_60. -1 unit 55m 48s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 19m 15s hadoop-common in the patch failed with JDK v1.7.0_79. -1 unit 57m 46s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 21s Patch generated 58 ASF License warnings. 187m 55s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   org.apache.hadoop.security.token.Token$PrivateToken doesn't override Token.equals(Object) At Token.java:At Token.java: [line 1] JDK v1.7.0_79 Failed junit tests hadoop.ipc.TestRPC   hadoop.ha.TestZKFailoverController   hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.TestNamenodeRetryCache   hadoop.hdfs.security.TestDelegationTokenForProxyUser JDK v1.7.0_79 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-29 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769535/HDFS-9276.07.patch JIRA Issue HDFS-9276 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux ac6e55b587f6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-67f42f1/precommit/personality/hadoop.sh git revision trunk / c416999 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13276/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13276/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 225MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13276/console This message was automatically generated.
          Hide
          hitliuyi Yi Liu added a comment -

          1. The code format is not correct. The indent is 2 spaces. If some line is too long and break it, then the indent is 4 spaces.
          2. You should cleanup the unused imports and whitespace.
          3. There is one findbugs, seems you need to override the equals in PrivateToken, and you can check whether the publicService is the same there.
          4.
          You can move your test into TestDelegationTokensWithHA, and you need to check access successful before cancel token1.

          Show
          hitliuyi Yi Liu added a comment - 1. The code format is not correct. The indent is 2 spaces. If some line is too long and break it, then the indent is 4 spaces. 2. You should cleanup the unused imports and whitespace. 3. There is one findbugs, seems you need to override the equals in PrivateToken, and you can check whether the publicService is the same there. 4. You can move your test into TestDelegationTokensWithHA , and you need to check access successful before cancel token1.
          Hide
          marsishandsome Liangliang Gu added a comment -

          I have to write the test case like this, otherwise the bug will not be reproduced.

          Show
          marsishandsome Liangliang Gu added a comment - I have to write the test case like this, otherwise the bug will not be reproduced.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 5s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 3s trunk passed
          +1 compile 4m 22s trunk passed with JDK v1.8.0_60
          +1 compile 4m 13s trunk passed with JDK v1.7.0_79
          +1 checkstyle 1m 2s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          -1 findbugs 2m 5s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 2m 5s trunk passed with JDK v1.8.0_60
          +1 javadoc 2m 57s trunk passed with JDK v1.7.0_79
          +1 mvninstall 2m 8s the patch passed
          +1 compile 4m 31s the patch passed with JDK v1.8.0_60
          +1 javac 4m 31s the patch passed
          +1 compile 4m 27s the patch passed with JDK v1.7.0_79
          +1 javac 4m 27s the patch passed
          -1 checkstyle 1m 1s Patch generated 1 new checkstyle issues in root (total was 29, now 30).
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 1m 58s hadoop-common-project/hadoop-common introduced 1 new FindBugs issues.
          +1 javadoc 2m 2s the patch passed with JDK v1.8.0_60
          +1 javadoc 2m 57s the patch passed with JDK v1.7.0_79
          -1 unit 6m 35s hadoop-common in the patch failed with JDK v1.8.0_60.
          -1 unit 55m 5s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          +1 unit 7m 20s hadoop-common in the patch passed with JDK v1.7.0_79.
          -1 unit 52m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 29s Patch generated 58 ASF License warnings.
          167m 22s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            org.apache.hadoop.security.token.Token$PrivateToken defines equals but not hashCode At Token.java:hashCode At Token.java:[lines 216-222]
          JDK v1.8.0_60 Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.hdfs.TestDFSClientRetries
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
            hadoop.fs.TestSymlinkHdfsFileContext
          JDK v1.7.0_79 Failed junit tests hadoop.fs.TestFcHdfsCreateMkdir



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-30
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769693/HDFS-9276.08.patch
          JIRA Issue HDFS-9276
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 4884109d65d4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build@2/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh
          git revision trunk / 7412ff4
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13295/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 227MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13295/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 5s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 3s trunk passed +1 compile 4m 22s trunk passed with JDK v1.8.0_60 +1 compile 4m 13s trunk passed with JDK v1.7.0_79 +1 checkstyle 1m 2s trunk passed +1 mvneclipse 0m 26s trunk passed -1 findbugs 2m 5s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 2m 5s trunk passed with JDK v1.8.0_60 +1 javadoc 2m 57s trunk passed with JDK v1.7.0_79 +1 mvninstall 2m 8s the patch passed +1 compile 4m 31s the patch passed with JDK v1.8.0_60 +1 javac 4m 31s the patch passed +1 compile 4m 27s the patch passed with JDK v1.7.0_79 +1 javac 4m 27s the patch passed -1 checkstyle 1m 1s Patch generated 1 new checkstyle issues in root (total was 29, now 30). +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 58s hadoop-common-project/hadoop-common introduced 1 new FindBugs issues. +1 javadoc 2m 2s the patch passed with JDK v1.8.0_60 +1 javadoc 2m 57s the patch passed with JDK v1.7.0_79 -1 unit 6m 35s hadoop-common in the patch failed with JDK v1.8.0_60. -1 unit 55m 5s hadoop-hdfs in the patch failed with JDK v1.8.0_60. +1 unit 7m 20s hadoop-common in the patch passed with JDK v1.7.0_79. -1 unit 52m 40s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 29s Patch generated 58 ASF License warnings. 167m 22s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   org.apache.hadoop.security.token.Token$PrivateToken defines equals but not hashCode At Token.java:hashCode At Token.java: [lines 216-222] JDK v1.8.0_60 Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery   hadoop.fs.TestSymlinkHdfsFileContext JDK v1.7.0_79 Failed junit tests hadoop.fs.TestFcHdfsCreateMkdir Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-30 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769693/HDFS-9276.08.patch JIRA Issue HDFS-9276 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 4884109d65d4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build@2/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh git revision trunk / 7412ff4 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13295/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13295/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13295/console This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          in the test, you don't need to catch that exception and convert to a fail(). Remove the try block and JUnit will handle it direclty

          Show
          stevel@apache.org Steve Loughran added a comment - in the test, you don't need to catch that exception and convert to a fail() . Remove the try block and JUnit will handle it direclty
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 6s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 2m 53s trunk passed
          +1 compile 4m 21s trunk passed with JDK v1.8.0_60
          +1 compile 4m 6s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 58s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          -1 findbugs 1m 50s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 1m 59s trunk passed with JDK v1.8.0_60
          +1 javadoc 2m 53s trunk passed with JDK v1.7.0_79
          +1 mvninstall 2m 1s the patch passed
          +1 compile 4m 19s the patch passed with JDK v1.8.0_60
          +1 javac 4m 19s the patch passed
          +1 compile 4m 10s the patch passed with JDK v1.7.0_79
          +1 javac 4m 10s the patch passed
          +1 checkstyle 0m 57s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 44s the patch passed
          +1 javadoc 2m 1s the patch passed with JDK v1.8.0_60
          +1 javadoc 2m 45s the patch passed with JDK v1.7.0_79
          -1 unit 6m 7s hadoop-common in the patch failed with JDK v1.8.0_60.
          -1 unit 51m 22s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          -1 unit 6m 26s hadoop-common in the patch failed with JDK v1.7.0_79.
          +1 unit 50m 2s hadoop-hdfs in the patch passed with JDK v1.7.0_79.
          -1 asflicense 0m 18s Patch generated 56 ASF License warnings.
          157m 0s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.ipc.TestDecayRpcScheduler
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
            hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.fs.contract.hdfs.TestHDFSContractConcat
            hadoop.hdfs.TestRecoverStripedFile
          JDK v1.7.0_79 Failed junit tests hadoop.metrics2.sink.TestFileSink



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-30
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769721/HDFS-9276.09.patch
          JIRA Issue HDFS-9276
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 297851a04413 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh
          git revision trunk / ce31b22
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13298/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13298/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13298/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13298/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13298/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13298/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13298/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13298/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13298/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 225MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13298/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 2m 53s trunk passed +1 compile 4m 21s trunk passed with JDK v1.8.0_60 +1 compile 4m 6s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 58s trunk passed +1 mvneclipse 0m 28s trunk passed -1 findbugs 1m 50s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 1m 59s trunk passed with JDK v1.8.0_60 +1 javadoc 2m 53s trunk passed with JDK v1.7.0_79 +1 mvninstall 2m 1s the patch passed +1 compile 4m 19s the patch passed with JDK v1.8.0_60 +1 javac 4m 19s the patch passed +1 compile 4m 10s the patch passed with JDK v1.7.0_79 +1 javac 4m 10s the patch passed +1 checkstyle 0m 57s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 44s the patch passed +1 javadoc 2m 1s the patch passed with JDK v1.8.0_60 +1 javadoc 2m 45s the patch passed with JDK v1.7.0_79 -1 unit 6m 7s hadoop-common in the patch failed with JDK v1.8.0_60. -1 unit 51m 22s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 6m 26s hadoop-common in the patch failed with JDK v1.7.0_79. +1 unit 50m 2s hadoop-hdfs in the patch passed with JDK v1.7.0_79. -1 asflicense 0m 18s Patch generated 56 ASF License warnings. 157m 0s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.ipc.TestDecayRpcScheduler   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.fs.contract.hdfs.TestHDFSContractConcat   hadoop.hdfs.TestRecoverStripedFile JDK v1.7.0_79 Failed junit tests hadoop.metrics2.sink.TestFileSink Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-30 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769721/HDFS-9276.09.patch JIRA Issue HDFS-9276 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 297851a04413 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh git revision trunk / ce31b22 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13298/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13298/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13298/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13298/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13298/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13298/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13298/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13298/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13298/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 225MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13298/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 8s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 43s trunk passed
          +1 compile 5m 22s trunk passed with JDK v1.8.0_60
          +1 compile 4m 50s trunk passed with JDK v1.7.0_79
          +1 checkstyle 1m 10s trunk passed
          +1 mvneclipse 0m 32s trunk passed
          -1 findbugs 2m 17s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 2m 30s trunk passed with JDK v1.8.0_60
          +1 javadoc 3m 19s trunk passed with JDK v1.7.0_79
          +1 mvninstall 2m 21s the patch passed
          +1 compile 5m 8s the patch passed with JDK v1.8.0_60
          +1 javac 5m 8s the patch passed
          +1 compile 4m 54s the patch passed with JDK v1.7.0_79
          +1 javac 4m 54s the patch passed
          +1 checkstyle 1m 15s the patch passed
          +1 mvneclipse 0m 32s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 41s the patch passed
          +1 javadoc 2m 7s the patch passed with JDK v1.8.0_60
          +1 javadoc 3m 4s the patch passed with JDK v1.7.0_79
          -1 unit 7m 16s hadoop-common in the patch failed with JDK v1.8.0_60.
          -1 unit 57m 9s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          +1 unit 7m 53s hadoop-common in the patch passed with JDK v1.7.0_79.
          -1 unit 55m 47s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 21s Patch generated 58 ASF License warnings.
          179m 47s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag
            hadoop.hdfs.TestDFSShell
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
            hadoop.hdfs.server.datanode.TestDataNodeMetrics
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-02
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770013/HDFS-9276.10.patch
          JIRA Issue HDFS-9276
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 9e02b2698ab5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh
          git revision trunk / 3cde693
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13320/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13320/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13320/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13320/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13320/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13320/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13320/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13320/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13320/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 228MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13320/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 8s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 43s trunk passed +1 compile 5m 22s trunk passed with JDK v1.8.0_60 +1 compile 4m 50s trunk passed with JDK v1.7.0_79 +1 checkstyle 1m 10s trunk passed +1 mvneclipse 0m 32s trunk passed -1 findbugs 2m 17s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 2m 30s trunk passed with JDK v1.8.0_60 +1 javadoc 3m 19s trunk passed with JDK v1.7.0_79 +1 mvninstall 2m 21s the patch passed +1 compile 5m 8s the patch passed with JDK v1.8.0_60 +1 javac 5m 8s the patch passed +1 compile 4m 54s the patch passed with JDK v1.7.0_79 +1 javac 4m 54s the patch passed +1 checkstyle 1m 15s the patch passed +1 mvneclipse 0m 32s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 41s the patch passed +1 javadoc 2m 7s the patch passed with JDK v1.8.0_60 +1 javadoc 3m 4s the patch passed with JDK v1.7.0_79 -1 unit 7m 16s hadoop-common in the patch failed with JDK v1.8.0_60. -1 unit 57m 9s hadoop-hdfs in the patch failed with JDK v1.8.0_60. +1 unit 7m 53s hadoop-common in the patch passed with JDK v1.7.0_79. -1 unit 55m 47s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 21s Patch generated 58 ASF License warnings. 179m 47s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag   hadoop.hdfs.TestDFSShell   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks   hadoop.hdfs.server.datanode.TestDataNodeMetrics JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-02 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770013/HDFS-9276.10.patch JIRA Issue HDFS-9276 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 9e02b2698ab5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh git revision trunk / 3cde693 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13320/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13320/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13320/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13320/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13320/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13320/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13320/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13320/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13320/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 228MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13320/console This message was automatically generated.
          Hide
          hitliuyi Yi Liu added a comment -

          Liangliang, thanks for the update. See following comments. (I forgot to add #1 in previous comments)
          1.

          tokenMap.put(alias, t);
          
          // update private token
          ...
          

          We can update the private token if the token exists, see following:

          if (tokenMap.put(alias, t) != null) {
            // update private token
            ...
          }
          

          2.

              @Override
              public int hashCode() {
                return super.hashCode() ^ publicService.hashCode();
              }
          

          Could be:

              @Override
              public int hashCode() {
                return super.hashCode();
              }
          

          3.
          I looked at testCancelAndUpdateDelegationTokens again, it doesn't work as expected. You added credentials for "Test", but use another user to do listFiles. Also only set DFSConfigKeys.DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY doesn't mean security enabled, and the authentication is still simple.

          Show
          hitliuyi Yi Liu added a comment - Liangliang, thanks for the update. See following comments. (I forgot to add #1 in previous comments) 1. tokenMap.put(alias, t); // update private token ... We can update the private token if the token exists, see following: if (tokenMap.put(alias, t) != null ) { // update private token ... } 2. @Override public int hashCode() { return super .hashCode() ^ publicService.hashCode(); } Could be: @Override public int hashCode() { return super .hashCode(); } 3. I looked at testCancelAndUpdateDelegationTokens again, it doesn't work as expected. You added credentials for "Test", but use another user to do listFiles. Also only set DFSConfigKeys.DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY doesn't mean security enabled, and the authentication is still simple.
          Hide
          marsishandsome Liangliang Gu added a comment -

          First of all, the new test is used to reproduce the bug this patch aims to fix.
          If you run the test without the patch, you will see the bug.

          I find that enabling kerberos is not a must for reproducing the bug.
          For simplicity, the test case does not enalbe kerberos.

          Show
          marsishandsome Liangliang Gu added a comment - First of all, the new test is used to reproduce the bug this patch aims to fix. If you run the test without the patch, you will see the bug. I find that enabling kerberos is not a must for reproducing the bug. For simplicity, the test case does not enalbe kerberos.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 8s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 17s trunk passed
          +1 compile 4m 52s trunk passed with JDK v1.8.0_60
          +1 compile 4m 30s trunk passed with JDK v1.7.0_79
          +1 checkstyle 1m 3s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          -1 findbugs 1m 58s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 2m 25s trunk passed with JDK v1.8.0_60
          +1 javadoc 3m 11s trunk passed with JDK v1.7.0_79
          +1 mvninstall 2m 10s the patch passed
          +1 compile 4m 40s the patch passed with JDK v1.8.0_60
          +1 javac 4m 40s the patch passed
          +1 compile 4m 30s the patch passed with JDK v1.7.0_79
          +1 javac 4m 30s the patch passed
          +1 checkstyle 1m 4s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 27s the patch passed
          +1 javadoc 2m 31s the patch passed with JDK v1.8.0_60
          +1 javadoc 3m 8s the patch passed with JDK v1.7.0_79
          +1 unit 7m 55s hadoop-common in the patch passed with JDK v1.8.0_60.
          -1 unit 58m 58s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          -1 unit 7m 15s hadoop-common in the patch failed with JDK v1.7.0_79.
          -1 unit 53m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 20s Patch generated 58 ASF License warnings.
          175m 45s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
          JDK v1.7.0_79 Failed junit tests hadoop.ipc.TestIPC
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
            hadoop.hdfs.server.datanode.TestBPOfferService



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-03
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770233/HDFS-9276.11.patch
          JIRA Issue HDFS-9276
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 96cf5cbd04a4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-1a9afee/precommit/personality/hadoop.sh
          git revision trunk / d565480
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13351/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13351/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13351/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13351/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13351/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13351/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13351/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13351/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13351/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 224MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13351/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 8s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 17s trunk passed +1 compile 4m 52s trunk passed with JDK v1.8.0_60 +1 compile 4m 30s trunk passed with JDK v1.7.0_79 +1 checkstyle 1m 3s trunk passed +1 mvneclipse 0m 28s trunk passed -1 findbugs 1m 58s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 2m 25s trunk passed with JDK v1.8.0_60 +1 javadoc 3m 11s trunk passed with JDK v1.7.0_79 +1 mvninstall 2m 10s the patch passed +1 compile 4m 40s the patch passed with JDK v1.8.0_60 +1 javac 4m 40s the patch passed +1 compile 4m 30s the patch passed with JDK v1.7.0_79 +1 javac 4m 30s the patch passed +1 checkstyle 1m 4s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 27s the patch passed +1 javadoc 2m 31s the patch passed with JDK v1.8.0_60 +1 javadoc 3m 8s the patch passed with JDK v1.7.0_79 +1 unit 7m 55s hadoop-common in the patch passed with JDK v1.8.0_60. -1 unit 58m 58s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 7m 15s hadoop-common in the patch failed with JDK v1.7.0_79. -1 unit 53m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 20s Patch generated 58 ASF License warnings. 175m 45s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 JDK v1.7.0_79 Failed junit tests hadoop.ipc.TestIPC   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.server.datanode.TestBPOfferService Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-03 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770233/HDFS-9276.11.patch JIRA Issue HDFS-9276 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 96cf5cbd04a4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-1a9afee/precommit/personality/hadoop.sh git revision trunk / d565480 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13351/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13351/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13351/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13351/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13351/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13351/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13351/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13351/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13351/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 224MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13351/console This message was automatically generated.
          Hide
          hitliuyi Yi Liu added a comment -

          Yes, even the security is not enabled, but if there is token in user's credentials, sasl is used, so will use delegation token for authentication.

          Further comments for tests:
          1. Currently we get token1 and token2 for current user, but create a "test" user to list files. We should make there consistent: either using doAs of "Test" to get tokens, or no need to create "Test" user, just use current user and add tokens to current user's UGI.
          2. no need to modify getDelegationToken, we can get the token1/token2 and add it to UGI directly, we also don't need creds1/creds2

          Show
          hitliuyi Yi Liu added a comment - Yes, even the security is not enabled, but if there is token in user's credentials, sasl is used, so will use delegation token for authentication. Further comments for tests: 1. Currently we get token1 and token2 for current user, but create a "test" user to list files. We should make there consistent: either using doAs of "Test" to get tokens, or no need to create "Test" user, just use current user and add tokens to current user's UGI. 2. no need to modify getDelegationToken , we can get the token1/token2 and add it to UGI directly, we also don't need creds1/creds2
          Hide
          marsishandsome Liangliang Gu added a comment -

          Thanks YiLiu

          Using

          UserGroupInformation.getCurrentUser().addToken

          can also reproduce this bug.

          The logical of this test case is copy from spark executor.
          When an executor is started in yarn, the current user already has a HDFS Delegation Token.
          Before the token is expired, the executor will download a new HDFS Delegation Token and update to the current user.

          The new user "test" simulates the user in yarn executor.
          Token1 simulates the token when the yarn executor has at start.
          Token2 simulates the token the yarn executor download.

          Show
          marsishandsome Liangliang Gu added a comment - Thanks YiLiu Using UserGroupInformation.getCurrentUser().addToken can also reproduce this bug. The logical of this test case is copy from spark executor. When an executor is started in yarn, the current user already has a HDFS Delegation Token. Before the token is expired, the executor will download a new HDFS Delegation Token and update to the current user. The new user "test" simulates the user in yarn executor. Token1 simulates the token when the yarn executor has at start. Token2 simulates the token the yarn executor download.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 35s trunk passed
          +1 compile 5m 30s trunk passed with JDK v1.8.0_60
          +1 compile 5m 7s trunk passed with JDK v1.7.0_79
          +1 checkstyle 1m 11s trunk passed
          +1 mvneclipse 0m 34s trunk passed
          -1 findbugs 2m 21s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings.
          +1 javadoc 2m 41s trunk passed with JDK v1.8.0_60
          +1 javadoc 3m 44s trunk passed with JDK v1.7.0_79
          +1 mvninstall 2m 39s the patch passed
          +1 compile 5m 29s the patch passed with JDK v1.8.0_60
          +1 javac 5m 29s the patch passed
          +1 compile 5m 6s the patch passed with JDK v1.7.0_79
          +1 javac 5m 6s the patch passed
          +1 checkstyle 1m 12s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 49s the patch passed
          +1 javadoc 2m 46s the patch passed with JDK v1.8.0_60
          +1 javadoc 3m 48s the patch passed with JDK v1.7.0_79
          -1 unit 8m 52s hadoop-common in the patch failed with JDK v1.8.0_60.
          -1 unit 74m 3s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          +1 unit 8m 57s hadoop-common in the patch passed with JDK v1.7.0_79.
          -1 unit 69m 16s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 25s Patch generated 56 ASF License warnings.
          216m 36s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.fs.shell.find.TestName
            hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.test.TestTimedOutTestsListener
            hadoop.hdfs.server.namenode.TestRecoverStripedBlocks
            hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestBlockStoragePolicy
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-06
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770938/HDFS-9276.12.patch
          JIRA Issue HDFS-9276
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux f14cabd6adcf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
          git revision trunk / bff7c90
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13414/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13414/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13414/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13414/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13414/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13414/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13414/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13414/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13414/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 228MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13414/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 35s trunk passed +1 compile 5m 30s trunk passed with JDK v1.8.0_60 +1 compile 5m 7s trunk passed with JDK v1.7.0_79 +1 checkstyle 1m 11s trunk passed +1 mvneclipse 0m 34s trunk passed -1 findbugs 2m 21s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings. +1 javadoc 2m 41s trunk passed with JDK v1.8.0_60 +1 javadoc 3m 44s trunk passed with JDK v1.7.0_79 +1 mvninstall 2m 39s the patch passed +1 compile 5m 29s the patch passed with JDK v1.8.0_60 +1 javac 5m 29s the patch passed +1 compile 5m 6s the patch passed with JDK v1.7.0_79 +1 javac 5m 6s the patch passed +1 checkstyle 1m 12s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 49s the patch passed +1 javadoc 2m 46s the patch passed with JDK v1.8.0_60 +1 javadoc 3m 48s the patch passed with JDK v1.7.0_79 -1 unit 8m 52s hadoop-common in the patch failed with JDK v1.8.0_60. -1 unit 74m 3s hadoop-hdfs in the patch failed with JDK v1.8.0_60. +1 unit 8m 57s hadoop-common in the patch passed with JDK v1.7.0_79. -1 unit 69m 16s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 25s Patch generated 56 ASF License warnings. 216m 36s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.fs.shell.find.TestName   hadoop.metrics2.impl.TestGangliaMetrics   hadoop.test.TestTimedOutTestsListener   hadoop.hdfs.server.namenode.TestRecoverStripedBlocks   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020   hadoop.hdfs.server.namenode.ha.TestEditLogTailer Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-06 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770938/HDFS-9276.12.patch JIRA Issue HDFS-9276 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux f14cabd6adcf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh git revision trunk / bff7c90 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13414/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13414/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13414/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13414/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13414/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13414/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13414/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13414/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13414/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 228MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13414/console This message was automatically generated.
          Hide
          hitliuyi Yi Liu added a comment -

          In the test, you get the token1/token2 for current login user, suppose it's UserA. When you use the token in other places, you should doAs that user "UserA", right? You doAs a different user "Test" (a remote login user), that means the service (spark executor) will use UserA's delegation token to access HDFS on behalf of "Test" user, is it right?

          Show
          hitliuyi Yi Liu added a comment - In the test, you get the token1/token2 for current login user, suppose it's UserA. When you use the token in other places, you should doAs that user "UserA", right? You doAs a different user "Test" (a remote login user), that means the service (spark executor) will use UserA's delegation token to access HDFS on behalf of "Test" user, is it right?
          Hide
          marsishandsome Liangliang Gu added a comment -

          I think Current User's Name instead of "Test" should be used.

          Show
          marsishandsome Liangliang Gu added a comment - I think Current User's Name instead of "Test" should be used.
          Hide
          marsishandsome Liangliang Gu added a comment -

          I think Current User's Name instead of "Test" should be used.

          Show
          marsishandsome Liangliang Gu added a comment - I think Current User's Name instead of "Test" should be used.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 8s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 17s trunk passed
          +1 compile 5m 3s trunk passed with JDK v1.8.0_66
          +1 compile 4m 38s trunk passed with JDK v1.7.0_79
          +1 checkstyle 1m 4s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          -1 findbugs 2m 8s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings.
          +1 javadoc 2m 16s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 12s trunk passed with JDK v1.7.0_79
          +1 mvninstall 2m 11s the patch passed
          +1 compile 4m 54s the patch passed with JDK v1.8.0_66
          +1 javac 4m 54s the patch passed
          +1 compile 4m 36s the patch passed with JDK v1.7.0_79
          +1 javac 4m 36s the patch passed
          +1 checkstyle 1m 5s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 17s the patch passed
          +1 javadoc 2m 22s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 6s the patch passed with JDK v1.7.0_79
          -1 unit 19m 32s hadoop-common in the patch failed with JDK v1.8.0_66.
          -1 unit 70m 25s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 8m 30s hadoop-common in the patch failed with JDK v1.7.0_79.
          -1 unit 70m 35s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 22s Patch generated 56 ASF License warnings.
          217m 47s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.server.datanode.TestTransferRbw
            hadoop.hdfs.TestDFSStripedOutputStream
            hadoop.hdfs.TestDistributedFileSystem
            hadoop.hdfs.server.datanode.TestBlockReplacement
            hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
          JDK v1.7.0_79 Failed junit tests hadoop.ipc.TestRPCWaitForProxy
            hadoop.hdfs.TestWriteReadStripedFile
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot



          Subsystem Report/Notes
          Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-06
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770987/HDFS-9276.13.patch
          JIRA Issue HDFS-9276
          Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
          uname Linux 1e099262c6b5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh
          git revision trunk / 66c0967
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13417/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 227MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13417/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 8s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 17s trunk passed +1 compile 5m 3s trunk passed with JDK v1.8.0_66 +1 compile 4m 38s trunk passed with JDK v1.7.0_79 +1 checkstyle 1m 4s trunk passed +1 mvneclipse 0m 29s trunk passed -1 findbugs 2m 8s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings. +1 javadoc 2m 16s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 12s trunk passed with JDK v1.7.0_79 +1 mvninstall 2m 11s the patch passed +1 compile 4m 54s the patch passed with JDK v1.8.0_66 +1 javac 4m 54s the patch passed +1 compile 4m 36s the patch passed with JDK v1.7.0_79 +1 javac 4m 36s the patch passed +1 checkstyle 1m 5s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 17s the patch passed +1 javadoc 2m 22s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 6s the patch passed with JDK v1.7.0_79 -1 unit 19m 32s hadoop-common in the patch failed with JDK v1.8.0_66. -1 unit 70m 25s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 8m 30s hadoop-common in the patch failed with JDK v1.7.0_79. -1 unit 70m 35s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 22s Patch generated 56 ASF License warnings. 217m 47s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.server.datanode.TestTransferRbw   hadoop.hdfs.TestDFSStripedOutputStream   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle JDK v1.7.0_79 Failed junit tests hadoop.ipc.TestRPCWaitForProxy   hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot Subsystem Report/Notes Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-06 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770987/HDFS-9276.13.patch JIRA Issue HDFS-9276 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 1e099262c6b5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-ee5baeb/precommit/personality/hadoop.sh git revision trunk / 66c0967 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13417/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13417/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13417/console This message was automatically generated.
          Hide
          daryn Daryn Sharp added a comment -

          I'll be taking a look to ensure this doesn't break our IP-failover HA.

          Show
          daryn Daryn Sharp added a comment - I'll be taking a look to ensure this doesn't break our IP-failover HA.
          Hide
          nielsbasjes Niels Basjes added a comment -

          I'm curious on the progress of this.
          This seems to be the reason our Apache Flink jobs fail after 7 days on our HA secured cluster.

          Show
          nielsbasjes Niels Basjes added a comment - I'm curious on the progress of this. This seems to be the reason our Apache Flink jobs fail after 7 days on our HA secured cluster.
          Hide
          jnp Jitendra Nath Pandey added a comment -

          Daryn Sharp are you planning to review this patch? The patch shouldn't impact IP-failover because the new private token preserves the service of the original private token.

          Show
          jnp Jitendra Nath Pandey added a comment - Daryn Sharp are you planning to review this patch? The patch shouldn't impact IP-failover because the new private token preserves the service of the original private token.
          Hide
          r1pp3rj4ck Attila Bukor added a comment -

          is there any update on this?

          Show
          r1pp3rj4ck Attila Bukor added a comment - is there any update on this?
          Hide
          hitliuyi Yi Liu added a comment -

          Not only one people tell me they encounter this issue in real cluster, and ask me to help pushing the fix.
          From my point of view, the approach in this patch is generally OK, and may still need some refinement.

          Daryn Sharp, Steve Loughran, Chris Nauroth, could you help to check too?

          Show
          hitliuyi Yi Liu added a comment - Not only one people tell me they encounter this issue in real cluster, and ask me to help pushing the fix. From my point of view, the approach in this patch is generally OK, and may still need some refinement. Daryn Sharp , Steve Loughran , Chris Nauroth , could you help to check too?
          Hide
          r1pp3rj4ck Attila Bukor added a comment -

          Daryn Sharp, Steve Loughran, Chris Nauroth did you have a chance to look into this issue yet?

          Show
          r1pp3rj4ck Attila Bukor added a comment - Daryn Sharp , Steve Loughran , Chris Nauroth did you have a chance to look into this issue yet?
          Hide
          svenkataraman666 Shankar Venkataraman added a comment -

          Is there any priority on this fix? We (Marketo) are running into this issue in Production in combination with SPARK-11182.

          Show
          svenkataraman666 Shankar Venkataraman added a comment - Is there any priority on this fix? We (Marketo) are running into this issue in Production in combination with SPARK-11182 .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 27s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 6m 41s trunk passed
          +1 compile 6m 53s trunk passed
          +1 checkstyle 1m 19s trunk passed
          +1 mvnsite 1m 49s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 5s trunk passed
          +1 javadoc 1m 43s trunk passed
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 1m 29s the patch passed
          +1 compile 6m 57s the patch passed
          +1 javac 6m 57s the patch passed
          -1 checkstyle 1m 21s root: The patch generated 4 new + 17 unchanged - 0 fixed = 21 total (was 17)
          +1 mvnsite 1m 51s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 17s the patch passed
          +1 javadoc 1m 52s the patch passed
          -1 unit 7m 40s hadoop-common in the patch failed.
          -1 unit 73m 44s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          122m 39s



          Reason Tests
          Failed junit tests hadoop.net.TestClusterTopology
            hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
            hadoop.hdfs.server.namenode.TestNamenodeRetryCache
            hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
            hadoop.hdfs.server.namenode.TestCacheDirectives
            hadoop.hdfs.TestEncryptionZonesWithKMS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e2f6409
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770987/HDFS-9276.13.patch
          JIRA Issue HDFS-9276
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6626635529ec 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8c1f81d
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15837/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15837/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15837/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15837/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15837/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15837/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15837/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 6m 41s trunk passed +1 compile 6m 53s trunk passed +1 checkstyle 1m 19s trunk passed +1 mvnsite 1m 49s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 5s trunk passed +1 javadoc 1m 43s trunk passed 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 29s the patch passed +1 compile 6m 57s the patch passed +1 javac 6m 57s the patch passed -1 checkstyle 1m 21s root: The patch generated 4 new + 17 unchanged - 0 fixed = 21 total (was 17) +1 mvnsite 1m 51s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 17s the patch passed +1 javadoc 1m 52s the patch passed -1 unit 7m 40s hadoop-common in the patch failed. -1 unit 73m 44s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 122m 39s Reason Tests Failed junit tests hadoop.net.TestClusterTopology   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.hdfs.server.namenode.TestNamenodeRetryCache   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.server.namenode.TestCacheDirectives   hadoop.hdfs.TestEncryptionZonesWithKMS Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770987/HDFS-9276.13.patch JIRA Issue HDFS-9276 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6626635529ec 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8c1f81d Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15837/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15837/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15837/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15837/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15837/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15837/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15837/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment - - edited

          I was able to reproduce with a simple long running Spark Scala application HDFSReadLoop.scala (attached) that performs HDFS read periodically. It hits the delegation toke expired exception after a little over 3 minutes. The cluster is HA with custom delegation properties specified in Description. HDFS-9276.13.patch does fix the issue.

          Show
          jzhuge John Zhuge added a comment - - edited I was able to reproduce with a simple long running Spark Scala application HDFSReadLoop.scala (attached) that performs HDFS read periodically. It hits the delegation toke expired exception after a little over 3 minutes. The cluster is HA with custom delegation properties specified in Description. HDFS-9276 .13.patch does fix the issue.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 14:

          • Minor code cleanups
          • Remove the use of instanceof

          The patch (applied on 2.6.0) passed 866 hadoop-common and hadoop-hdfs unit tests.

          Show
          jzhuge John Zhuge added a comment - Patch 14: Minor code cleanups Remove the use of instanceof The patch (applied on 2.6.0) passed 866 hadoop-common and hadoop-hdfs unit tests.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 58s trunk passed
          +1 compile 7m 10s trunk passed
          +1 checkstyle 1m 24s trunk passed
          +1 mvnsite 1m 52s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 3s trunk passed
          +1 javadoc 1m 42s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 31s the patch passed
          +1 compile 7m 20s the patch passed
          +1 javac 7m 20s the patch passed
          +1 checkstyle 1m 24s the patch passed
          +1 mvnsite 1m 52s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 24s the patch passed
          +1 javadoc 1m 43s the patch passed
          -1 unit 7m 4s hadoop-common in the patch failed.
          +1 unit 56m 22s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          105m 45s



          Reason Tests
          Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819621/HDFS-9276.14.patch
          JIRA Issue HDFS-9276
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 001d18e22228 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / dca6335
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16158/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16158/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16158/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 58s trunk passed +1 compile 7m 10s trunk passed +1 checkstyle 1m 24s trunk passed +1 mvnsite 1m 52s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 3s trunk passed +1 javadoc 1m 42s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 31s the patch passed +1 compile 7m 20s the patch passed +1 javac 7m 20s the patch passed +1 checkstyle 1m 24s the patch passed +1 mvnsite 1m 52s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 24s the patch passed +1 javadoc 1m 43s the patch passed -1 unit 7m 4s hadoop-common in the patch failed. +1 unit 56m 22s hadoop-hdfs in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 105m 45s Reason Tests Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819621/HDFS-9276.14.patch JIRA Issue HDFS-9276 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 001d18e22228 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dca6335 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16158/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16158/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16158/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Ignore the unrelated TestGangliaMetrics failure. What other tests should we run?

          +1 (non-binding) Looks good to me. Thanks Liangliang Gu for working on the patch, Yi Liu for the reviews. Could someone else please take a look? This issue has been hit by many customers.

          Show
          jzhuge John Zhuge added a comment - Ignore the unrelated TestGangliaMetrics failure. What other tests should we run? +1 (non-binding) Looks good to me. Thanks Liangliang Gu for working on the patch, Yi Liu for the reviews. Could someone else please take a look? This issue has been hit by many customers.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 15:

          • Revert back to getPublicService API that is more flexible than isPrivateClone and equally clean
          • To test a private clone: alias.equals(token.getPublicService())
          • To test a private token: token.getPublicService() == null
          Show
          jzhuge John Zhuge added a comment - Patch 15: Revert back to getPublicService API that is more flexible than isPrivateClone and equally clean To test a private clone: alias.equals(token.getPublicService()) To test a private token: token.getPublicService() == null
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 8m 0s Maven dependency ordering for branch
          +1 mvninstall 8m 0s trunk passed
          +1 compile 8m 58s trunk passed
          +1 checkstyle 1m 29s trunk passed
          +1 mvnsite 2m 7s trunk passed
          +1 mvneclipse 0m 31s trunk passed
          +1 findbugs 3m 54s trunk passed
          +1 javadoc 2m 1s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 57s the patch passed
          +1 compile 7m 14s the patch passed
          -1 javac 7m 14s root generated 1 new + 708 unchanged - 1 fixed = 709 total (was 709)
          +1 checkstyle 1m 30s the patch passed
          +1 mvnsite 1m 56s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 21s the patch passed
          +1 javadoc 1m 49s the patch passed
          -1 unit 19m 31s hadoop-common in the patch failed.
          -1 unit 90m 0s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          164m 53s



          Reason Tests
          Failed junit tests hadoop.security.TestUserGroupInformation
            hadoop.hdfs.security.TestDelegationTokenForProxyUser
            hadoop.hdfs.server.datanode.TestDataNodeLifeline
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
          Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
            org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819774/HDFS-9276.15.patch
          JIRA Issue HDFS-9276
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b852d404bc6e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b79ba4f
          Default Java 1.8.0_91
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/16163/artifact/patchprocess/diff-compile-javac-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16163/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16163/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16163/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16163/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 8m 0s Maven dependency ordering for branch +1 mvninstall 8m 0s trunk passed +1 compile 8m 58s trunk passed +1 checkstyle 1m 29s trunk passed +1 mvnsite 2m 7s trunk passed +1 mvneclipse 0m 31s trunk passed +1 findbugs 3m 54s trunk passed +1 javadoc 2m 1s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 57s the patch passed +1 compile 7m 14s the patch passed -1 javac 7m 14s root generated 1 new + 708 unchanged - 1 fixed = 709 total (was 709) +1 checkstyle 1m 30s the patch passed +1 mvnsite 1m 56s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 21s the patch passed +1 javadoc 1m 49s the patch passed -1 unit 19m 31s hadoop-common in the patch failed. -1 unit 90m 0s hadoop-hdfs in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 164m 53s Reason Tests Failed junit tests hadoop.security.TestUserGroupInformation   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.datanode.TestDataNodeLifeline   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle   org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819774/HDFS-9276.15.patch JIRA Issue HDFS-9276 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b852d404bc6e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b79ba4f Default Java 1.8.0_91 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/16163/artifact/patchprocess/diff-compile-javac-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16163/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16163/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16163/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16163/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment - - edited

          Patch 16:

          • Fix test TestUserGroupInformation failure
          • Fix javac unchecked warning by replacing new Token.PrivateToken(t) with new Token.PrivateToken<>(t). No need to suppress the warning.
            [unchecked] unchecked call to PrivateToken(Token<T>) as a member of the raw type PrivateToken
            
          • Pass these unit tests: TestUserGroupInformation,TestDelegationTokenForProxyUser,TestDataNodeLifeline,TestCrcCorruption,TestBlockTokenWithDFSStriped,TestHttpServerLifecycle,TestLeaseRecovery2
          Show
          jzhuge John Zhuge added a comment - - edited Patch 16: Fix test TestUserGroupInformation failure Fix javac unchecked warning by replacing new Token.PrivateToken(t) with new Token.PrivateToken<>(t) . No need to suppress the warning. [unchecked] unchecked call to PrivateToken(Token<T>) as a member of the raw type PrivateToken Pass these unit tests: TestUserGroupInformation,TestDelegationTokenForProxyUser,TestDataNodeLifeline,TestCrcCorruption,TestBlockTokenWithDFSStriped,TestHttpServerLifecycle,TestLeaseRecovery2
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 6m 53s trunk passed
          +1 compile 6m 52s trunk passed
          +1 checkstyle 1m 22s trunk passed
          +1 mvnsite 1m 47s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 2s trunk passed
          +1 javadoc 1m 40s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 27s the patch passed
          +1 compile 6m 48s the patch passed
          +1 javac 6m 48s the patch passed
          +1 checkstyle 1m 22s the patch passed
          +1 mvnsite 1m 44s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 17s the patch passed
          +1 javadoc 1m 43s the patch passed
          -1 unit 19m 26s hadoop-common in the patch failed.
          +1 unit 68m 44s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          129m 0s



          Reason Tests
          Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819788/HDFS-9276.16.patch
          JIRA Issue HDFS-9276
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d9a730710f30 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b79ba4f
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16165/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16165/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16165/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 6m 53s trunk passed +1 compile 6m 52s trunk passed +1 checkstyle 1m 22s trunk passed +1 mvnsite 1m 47s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 2s trunk passed +1 javadoc 1m 40s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 6m 48s the patch passed +1 javac 6m 48s the patch passed +1 checkstyle 1m 22s the patch passed +1 mvnsite 1m 44s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 17s the patch passed +1 javadoc 1m 43s the patch passed -1 unit 19m 26s hadoop-common in the patch failed. +1 unit 68m 44s hadoop-hdfs in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 129m 0s Reason Tests Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819788/HDFS-9276.16.patch JIRA Issue HDFS-9276 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d9a730710f30 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b79ba4f Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16165/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16165/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16165/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          All unit tests seemed to have passed:

          Tests run: 3462, Failures: 0, Errors: 0, Skipped: 84
          
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 19:18.435s
          [INFO] Finished at: Sat Jul 23 17:38:22 UTC 2016
          [INFO] Final Memory: 25M/302M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-common:
           There was a timeout or other error in the fork -> [Help 1]
          [ERROR] 
          
          Show
          jzhuge John Zhuge added a comment - All unit tests seemed to have passed: Tests run: 3462, Failures: 0, Errors: 0, Skipped: 84 [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 19:18.435s [INFO] Finished at: Sat Jul 23 17:38:22 UTC 2016 [INFO] Final Memory: 25M/302M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-common: There was a timeout or other error in the fork -> [Help 1] [ERROR]
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks all for the contribution together!

          I'm catching up on this one, and below my understanding, is it correct?
          The problem is for HDFS-HA tokens, we use Token$PrivateToken to control what's visible. But currently clients doesn't have a way to update the private tokens when they need (e.g. when they found the current token is about to exceed lifetime, and gets a new one). And this jira adds the ability to update the private token.

          Seems Liangliang Gu and Yi Liu has done the major amount of work, and John Zhuge is trying to help finalizing it.

          I also have some comments on patch 16, mostly nits:
          Crendentials:

          • We combined the check on tokenMap.put not returning null with the parameter t's nullity check. Personally prefer to keep the tokenMap.put check inside. Otherwise we'll get confusing WARN logs.
          • alias's null check is done in the for loop. Let's move it up to the entrance.

          Token:

          • The new publicService field can be final
          • IMHO the 'public service' concept is only meaningful for PrivateToken, so the following in Token would be confusing. I think a typecast is more readable for future.
              /**
               * Return the public service.
               * @return null for non-private tokens
               */
              public Text getPublicService() {
                return null;
              }
            

          Test:

          • Yi's comment can be further addressed.

            In the test, you get the token1/token2 for current login user, suppose it's UserA. When you use the token in other places, you should doAs that user "UserA", right? You doAs a different user "Test" (a remote login user), that means the service (spark executor) will use UserA's delegation token to access HDFS on behalf of "Test" user, is it right?

            Since we're just testing delegation tokens here, can we create a whatever ugi, and get token1/token2 (and all current test codes) inside the doAs?

          Show
          xiaochen Xiao Chen added a comment - Thanks all for the contribution together! I'm catching up on this one, and below my understanding, is it correct? The problem is for HDFS-HA tokens, we use Token$PrivateToken to control what's visible. But currently clients doesn't have a way to update the private tokens when they need (e.g. when they found the current token is about to exceed lifetime, and gets a new one). And this jira adds the ability to update the private token. Seems Liangliang Gu and Yi Liu has done the major amount of work, and John Zhuge is trying to help finalizing it. I also have some comments on patch 16, mostly nits: Crendentials: We combined the check on tokenMap.put not returning null with the parameter t 's nullity check. Personally prefer to keep the tokenMap.put check inside. Otherwise we'll get confusing WARN logs. alias 's null check is done in the for loop. Let's move it up to the entrance. Token: The new publicService field can be final IMHO the 'public service' concept is only meaningful for PrivateToken , so the following in Token would be confusing. I think a typecast is more readable for future. /** * Return the public service. * @ return null for non- private tokens */ public Text getPublicService() { return null ; } Test: Yi's comment can be further addressed. In the test, you get the token1/token2 for current login user, suppose it's UserA. When you use the token in other places, you should doAs that user "UserA", right? You doAs a different user "Test" (a remote login user), that means the service (spark executor) will use UserA's delegation token to access HDFS on behalf of "Test" user, is it right? Since we're just testing delegation tokens here, can we create a whatever ugi, and get token1/token2 (and all current test codes) inside the doAs?
          Hide
          jzhuge John Zhuge added a comment -

          Thanks Xiao Chen for the great comments. Another patch is coming with the fixes.

          I discover a bug in Patch 13:

             public PrivateToken(Token<T> token) {
               super(token);
               publicService = token.getService();
             }
          

          publicService will point to the same object as token.service. Whenever the content of token.service is changed, publicService will reflect the change. I don't think this is what we want. It should perform a deep copy:

             public PrivateToken(Token<T> token) {
               super(token);
               publicService = new Text(token.getService());
             }
          

          IMHO the 'public service' concept is only meaningful for PrivateToken, so the following in Token would be confusing. I think a typecast is more readable for future.

          I'd avoid instanceof or typecasting. Whenever the need for that kind of code pops up, it indicates something not right in the class design. However, this jira is not the right place for refactoring. I will file a separate jira to:

          • Avoid instanceof or typecasting of Toke.PrivateToken with an interface method technique
          • Make class Toke.PrivateToken private. Use a factory method instead.
          Show
          jzhuge John Zhuge added a comment - Thanks Xiao Chen for the great comments. Another patch is coming with the fixes. I discover a bug in Patch 13: public PrivateToken(Token<T> token) { super (token); publicService = token.getService(); } publicService will point to the same object as token.service . Whenever the content of token.service is changed, publicService will reflect the change. I don't think this is what we want. It should perform a deep copy: public PrivateToken(Token<T> token) { super (token); publicService = new Text(token.getService()); } IMHO the 'public service' concept is only meaningful for PrivateToken, so the following in Token would be confusing. I think a typecast is more readable for future. I'd avoid instanceof or typecasting. Whenever the need for that kind of code pops up, it indicates something not right in the class design. However, this jira is not the right place for refactoring. I will file a separate jira to: Avoid instanceof or typecasting of Toke.PrivateToken with an interface method technique Make class Toke.PrivateToken private. Use a factory method instead.
          Hide
          jzhuge John Zhuge added a comment - - edited

          Another thing that bothers me is Token copy constructor performs shallow copy of all 4 fields. Is this by design? Could Jitendra Nath Pandey or Owen O'Malley please comment since it is added by MAPREDUCE-2764?

           public Token(Token<T> other) {
              this.identifier = other.identifier;           <<< byte[]
              this.password = other.password;               <<< byte[]
              this.kind = other.kind;                       <<< Text
              this.service = other.service;                 <<< Text
            }
          
          Show
          jzhuge John Zhuge added a comment - - edited Another thing that bothers me is Token copy constructor performs shallow copy of all 4 fields. Is this by design? Could Jitendra Nath Pandey or Owen O'Malley please comment since it is added by MAPREDUCE-2764 ? public Token(Token<T> other) { this .identifier = other.identifier; <<< byte [] this .password = other.password; <<< byte [] this .kind = other.kind; <<< Text this .service = other.service; <<< Text }
          Hide
          jzhuge John Zhuge added a comment -

          Patch 17:

          • Incorporate review comments
          • Make a deep copy of token.getService() to publicService
          • Revert Credentials.addToken back to use instanceof Token.PrivateToken and typecasting as in Patch 13
          Show
          jzhuge John Zhuge added a comment - Patch 17: Incorporate review comments Make a deep copy of token.getService() to publicService Revert Credentials.addToken back to use instanceof Token.PrivateToken and typecasting as in Patch 13
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 46s Maven dependency ordering for branch
          +1 mvninstall 6m 44s trunk passed
          +1 compile 6m 54s trunk passed
          +1 checkstyle 1m 26s trunk passed
          +1 mvnsite 1m 47s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 2s trunk passed
          +1 javadoc 1m 39s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 26s the patch passed
          +1 compile 7m 15s the patch passed
          +1 javac 7m 15s the patch passed
          +1 checkstyle 1m 30s the patch passed
          +1 mvnsite 1m 59s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 30s the patch passed
          +1 javadoc 1m 52s the patch passed
          -1 unit 20m 27s hadoop-common in the patch failed.
          -1 unit 79m 5s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          141m 52s



          Reason Tests
          Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager
            hadoop.hdfs.server.namenode.TestEditLog
          Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
            org.apache.hadoop.hdfs.server.namenode.ha.TestHAAppend



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819842/HDFS-9276.17.patch
          JIRA Issue HDFS-9276
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 69a02910fecf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7052ca8
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16167/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16167/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16167/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16167/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 46s Maven dependency ordering for branch +1 mvninstall 6m 44s trunk passed +1 compile 6m 54s trunk passed +1 checkstyle 1m 26s trunk passed +1 mvnsite 1m 47s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 2s trunk passed +1 javadoc 1m 39s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 26s the patch passed +1 compile 7m 15s the patch passed +1 javac 7m 15s the patch passed +1 checkstyle 1m 30s the patch passed +1 mvnsite 1m 59s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 30s the patch passed +1 javadoc 1m 52s the patch passed -1 unit 20m 27s hadoop-common in the patch failed. -1 unit 79m 5s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 141m 52s Reason Tests Failed junit tests hadoop.security.ssl.TestReloadingX509TrustManager   hadoop.hdfs.server.namenode.TestEditLog Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle   org.apache.hadoop.hdfs.server.namenode.ha.TestHAAppend Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819842/HDFS-9276.17.patch JIRA Issue HDFS-9276 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 69a02910fecf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7052ca8 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16167/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16167/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16167/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16167/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Hi Liangliang Gu, hope you don't mind we have been pushing this jira forward. A customer is in urgent need of a patch.

          Could you or Yi Liu explain why PrivateToken#publicService was added? Was it because PrivateToken.getService() could not be trusted? If so, can we solve it by deep-copying service in Token copy constructor?

           public Token(Token<T> other) {
              this.identifier = other.identifier;
              this.password = other.password;
              this.kind = other.kind;
              this.service = new Text(other.service);
            }
          

          Or even deep-copying every field?

           public Token(Token<T> other) {
              this.identifier = other.identifier.clone();
              this.password = other.password.clone();
              this.kind = new Text(other.kind);
              this.service = new Text(other.service);
            }
          
          Show
          jzhuge John Zhuge added a comment - Hi Liangliang Gu , hope you don't mind we have been pushing this jira forward. A customer is in urgent need of a patch. Could you or Yi Liu explain why PrivateToken#publicService was added? Was it because PrivateToken.getService() could not be trusted? If so, can we solve it by deep-copying service in Token copy constructor? public Token(Token<T> other) { this .identifier = other.identifier; this .password = other.password; this .kind = other.kind; this .service = new Text(other.service); } Or even deep-copying every field? public Token(Token<T> other) { this .identifier = other.identifier.clone(); this .password = other.password.clone(); this .kind = new Text(other.kind); this .service = new Text(other.service); }
          Hide
          marsishandsome Liangliang Gu added a comment -

          Hi John,

          Token's field serivce maybe changed after calling Credentials:: addToken.
          So I suggest to use deep-copying.

          Show
          marsishandsome Liangliang Gu added a comment - Hi John, Token's field serivce maybe changed after calling Credentials:: addToken. So I suggest to use deep-copying.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 18:

          • Change Token copy constructor to do deep copies.
          • No need for the extra field PrivateToken#publicService, thus no change to PrivateToken at all.
          • Change Credentials.addToken to use getService().
          Show
          jzhuge John Zhuge added a comment - Patch 18: Change Token copy constructor to do deep copies. No need for the extra field PrivateToken#publicService , thus no change to PrivateToken at all. Change Credentials.addToken to use getService() .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 13m 8s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 5m 32s Maven dependency ordering for branch
          +1 mvninstall 8m 11s trunk passed
          +1 compile 7m 31s trunk passed
          +1 checkstyle 1m 31s trunk passed
          +1 mvnsite 2m 0s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 3m 25s trunk passed
          +1 javadoc 1m 44s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 31s the patch passed
          +1 compile 7m 13s the patch passed
          +1 javac 7m 13s the patch passed
          +1 checkstyle 1m 23s the patch passed
          +1 mvnsite 1m 51s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 26s the patch passed
          +1 javadoc 1m 42s the patch passed
          +1 unit 7m 43s hadoop-common in the patch passed.
          -1 unit 60m 11s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          130m 28s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820037/HDFS-9276.18.patch
          JIRA Issue HDFS-9276
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7b859c3b5844 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 703fdf8
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16176/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16176/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16176/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 13m 8s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 5m 32s Maven dependency ordering for branch +1 mvninstall 8m 11s trunk passed +1 compile 7m 31s trunk passed +1 checkstyle 1m 31s trunk passed +1 mvnsite 2m 0s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 3m 25s trunk passed +1 javadoc 1m 44s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 31s the patch passed +1 compile 7m 13s the patch passed +1 javac 7m 13s the patch passed +1 checkstyle 1m 23s the patch passed +1 mvnsite 1m 51s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 26s the patch passed +1 javadoc 1m 42s the patch passed +1 unit 7m 43s hadoop-common in the patch passed. -1 unit 60m 11s hadoop-hdfs in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 130m 28s Reason Tests Failed junit tests hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820037/HDFS-9276.18.patch JIRA Issue HDFS-9276 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7b859c3b5844 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 703fdf8 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16176/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16176/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16176/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Test error in TestBalancerWithSaslDataTransfer.testBalancer0Integrity not related.

          Show
          jzhuge John Zhuge added a comment - Test error in TestBalancerWithSaslDataTransfer.testBalancer0Integrity not related.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks John Zhuge for revving.

          As we talked offline, fixing the copy constructor to do deep-copying feels to be the reasonable thing to do here. And the change looks cleaner.

          Nits:

          • In Credentials#addToken, let's keep the nullity check on alias.
          • We don't need the type cast to PrivateToken when checking token.getService().equals(alias).

          Also, I had a quick try locally, the added test case seems to be passing without any fix.... could you take a further look?

          Show
          xiaochen Xiao Chen added a comment - Thanks John Zhuge for revving. As we talked offline, fixing the copy constructor to do deep-copying feels to be the reasonable thing to do here. And the change looks cleaner. Nits: In Credentials#addToken , let's keep the nullity check on alias. We don't need the type cast to PrivateToken when checking token.getService().equals(alias) . Also, I had a quick try locally, the added test case seems to be passing without any fix.... could you take a further look?
          Hide
          jzhuge John Zhuge added a comment -

          Xiao Chen, Thanks for the catch on the unit test! Patch 17 introduced the regression.

          Show
          jzhuge John Zhuge added a comment - Xiao Chen , Thanks for the catch on the unit test! Patch 17 introduced the regression.
          Hide
          jzhuge John Zhuge added a comment -

          And because of the wrong unit test, changes in 17 and 18 were not properly validated. It looks like publicService is needed for PrivateToken because it does need 2 services: 1 for lookup and 1 for refreshing private tokens.

          Learning from the mistake, I will test the next patch against my Spark program HDFSReadLoop.scala in addition to the unit test.

          Show
          jzhuge John Zhuge added a comment - And because of the wrong unit test, changes in 17 and 18 were not properly validated. It looks like publicService is needed for PrivateToken because it does need 2 services: 1 for lookup and 1 for refreshing private tokens. Learning from the mistake, I will test the next patch against my Spark program HDFSReadLoop.scala in addition to the unit test.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 19:

          • Fix the unit test bug
          • Bring back publicService related code

          Passed units.
          Passed HDFSReadLoop spark test.

          Show
          jzhuge John Zhuge added a comment - Patch 19: Fix the unit test bug Bring back publicService related code Passed units. Passed HDFSReadLoop spark test.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 48m 49s Maven dependency ordering for branch
          +1 mvninstall 8m 2s trunk passed
          +1 compile 7m 1s trunk passed
          +1 checkstyle 1m 25s trunk passed
          +1 mvnsite 1m 57s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 2s trunk passed
          +1 javadoc 1m 40s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 27s the patch passed
          +1 compile 6m 50s the patch passed
          +1 javac 6m 50s the patch passed
          +1 checkstyle 1m 25s the patch passed
          +1 mvnsite 1m 46s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 28s the patch passed
          +1 javadoc 1m 44s the patch passed
          -1 unit 21m 25s hadoop-common in the patch failed.
          -1 unit 60m 51s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          173m 22s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.balancer.TestBalancer
          Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820109/HDFS-9276.19.patch
          JIRA Issue HDFS-9276
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9cb20ebc044a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7cac765
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16186/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16186/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16186/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16186/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 48m 49s Maven dependency ordering for branch +1 mvninstall 8m 2s trunk passed +1 compile 7m 1s trunk passed +1 checkstyle 1m 25s trunk passed +1 mvnsite 1m 57s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 2s trunk passed +1 javadoc 1m 40s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 6m 50s the patch passed +1 javac 6m 50s the patch passed +1 checkstyle 1m 25s the patch passed +1 mvnsite 1m 46s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 28s the patch passed +1 javadoc 1m 44s the patch passed -1 unit 21m 25s hadoop-common in the patch failed. -1 unit 60m 51s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 173m 22s Reason Tests Failed junit tests hadoop.hdfs.server.balancer.TestBalancer Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820109/HDFS-9276.19.patch JIRA Issue HDFS-9276 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9cb20ebc044a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7cac765 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16186/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16186/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16186/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16186/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          TestBalancer and TestHttpServerLifecycle failures areunrelated and they pass on my local host.

          Show
          jzhuge John Zhuge added a comment - TestBalancer and TestHttpServerLifecycle failures areunrelated and they pass on my local host.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks John Zhuge, patch 19 LGTM.
          I only have 1 nit: Text key = e.getKey(); can be done later, right before the put and inside the if clause.

          +1 after that's addressed.

          Since there's a big audience here, I'll let this one float for 2 days.
          Yi Liu, Daryn Sharp, Steve Loughran, Liangliang Gu, could you please take a look if you have time? I plan to commit this by end of Thursday if no more comments by then.

          Show
          xiaochen Xiao Chen added a comment - Thanks John Zhuge , patch 19 LGTM. I only have 1 nit: Text key = e.getKey(); can be done later, right before the put and inside the if clause. +1 after that's addressed. Since there's a big audience here, I'll let this one float for 2 days. Yi Liu , Daryn Sharp , Steve Loughran , Liangliang Gu , could you please take a look if you have time? I plan to commit this by end of Thursday if no more comments by then.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 20:

          • Small change in Credentials.addToken

          Thanks Xiao Chen.

          Show
          jzhuge John Zhuge added a comment - Patch 20: Small change in Credentials.addToken Thanks Xiao Chen .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 11m 58s Maven dependency ordering for branch
          +1 mvninstall 7m 16s trunk passed
          +1 compile 6m 48s trunk passed
          +1 checkstyle 1m 28s trunk passed
          +1 mvnsite 1m 50s trunk passed
          +1 mvneclipse 0m 30s trunk passed
          +1 findbugs 3m 20s trunk passed
          +1 javadoc 1m 41s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 28s the patch passed
          +1 compile 8m 11s the patch passed
          +1 javac 8m 11s the patch passed
          +1 checkstyle 1m 31s the patch passed
          +1 mvnsite 1m 57s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 45s the patch passed
          +1 javadoc 1m 46s the patch passed
          +1 unit 9m 7s hadoop-common in the patch passed.
          -1 unit 61m 24s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          126m 14s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestEditLog
            hadoop.cli.TestHDFSCLI



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820335/HDFS-9276.20.patch
          JIRA Issue HDFS-9276
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5e08d74286f1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d84ab8a
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16203/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16203/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16203/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 11m 58s Maven dependency ordering for branch +1 mvninstall 7m 16s trunk passed +1 compile 6m 48s trunk passed +1 checkstyle 1m 28s trunk passed +1 mvnsite 1m 50s trunk passed +1 mvneclipse 0m 30s trunk passed +1 findbugs 3m 20s trunk passed +1 javadoc 1m 41s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 8m 11s the patch passed +1 javac 8m 11s the patch passed +1 checkstyle 1m 31s the patch passed +1 mvnsite 1m 57s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 45s the patch passed +1 javadoc 1m 46s the patch passed +1 unit 9m 7s hadoop-common in the patch passed. -1 unit 61m 24s hadoop-hdfs in the patch failed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 126m 14s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestEditLog   hadoop.cli.TestHDFSCLI Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820335/HDFS-9276.20.patch JIRA Issue HDFS-9276 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5e08d74286f1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d84ab8a Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16203/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16203/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16203/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          amitsela Amit Sela added a comment -
          Show
          amitsela Amit Sela added a comment - Is this related to: https://issues.apache.org/jira/browse/YARN-896 ?
          Hide
          vinayrpet Vinayakumar B added a comment -

          Changes looks good to me as well. +1.

          Show
          vinayrpet Vinayakumar B added a comment - Changes looks good to me as well. +1.
          Hide
          jzhuge John Zhuge added a comment -

          Partially related. The patch fixes HDFS delegation token renewal bug In HA mode.

          Show
          jzhuge John Zhuge added a comment - Partially related. The patch fixes HDFS delegation token renewal bug In HA mode.
          Hide
          xiaochen Xiao Chen added a comment -

          +1 on patch 20, will commit shortly.

          Show
          xiaochen Xiao Chen added a comment - +1 on patch 20, will commit shortly.
          Hide
          xiaochen Xiao Chen added a comment -

          Failed tests looks unrelated, and passed locally. (TestHDFSCLI was recently fixed by HDFS-10696).

          Committed to trunk and branch-2. Thanks Liangliang Gu, John Zhuge for working on this, and Yi Liu, Steve Loughran and Vinayakumar B for the reviews, and everyone for the comments!

          Show
          xiaochen Xiao Chen added a comment - Failed tests looks unrelated, and passed locally. (TestHDFSCLI was recently fixed by HDFS-10696 ). Committed to trunk and branch-2. Thanks Liangliang Gu , John Zhuge for working on this, and Yi Liu , Steve Loughran and Vinayakumar B for the reviews, and everyone for the comments!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10174 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10174/)
          HDFS-9276. Failed to Update HDFS Delegation Token for long running (xiao: rev d9aae22fdf2ab22ae8ce4a9d32ac71b3dde084d3)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10174 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10174/ ) HDFS-9276 . Failed to Update HDFS Delegation Token for long running (xiao: rev d9aae22fdf2ab22ae8ce4a9d32ac71b3dde084d3) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java
          Hide
          jzhuge John Zhuge added a comment -

          Thanks Xiao Chen!

          Show
          jzhuge John Zhuge added a comment - Thanks Xiao Chen !
          Hide
          xiaochen Xiao Chen added a comment -

          Linking HADOOP-8751. Without it, TestUserGroupInformation#testPrivateTokenExclusion will fail with NPE.

          Show
          xiaochen Xiao Chen added a comment - Linking HADOOP-8751 . Without it, TestUserGroupInformation#testPrivateTokenExclusion will fail with NPE.
          Hide
          mariakatosvich mariakatosvich (Inactive) added a comment -

          In long running applications, the tokens will be expired, so we must update the tokens.
          TestBalancer and TestHttpServerLifecycle failures areunrelated and they pass on my local host.
          Thanks
          http://qwikfix.co.uk/hmrc-contact-number/

          Show
          mariakatosvich mariakatosvich (Inactive) added a comment - In long running applications, the tokens will be expired, so we must update the tokens. TestBalancer and TestHttpServerLifecycle failures areunrelated and they pass on my local host. Thanks http://qwikfix.co.uk/hmrc-contact-number/
          Hide
          jzhuge John Zhuge added a comment -

          The previous comment seems to be a spam, repeating some of our old comments. Do not click any link!

          Show
          jzhuge John Zhuge added a comment - The previous comment seems to be a spam, repeating some of our old comments. Do not click any link!

            People

            • Assignee:
              marsishandsome Liangliang Gu
              Reporter:
              marsishandsome Liangliang Gu
            • Votes:
              8 Vote for this issue
              Watchers:
              46 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development