Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: documentation
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Some Kerberos configuration parameters are not documented well enough.

      1. HDFS-9254.01.patch
        37 kB
        Arpit Agarwal
      2. HDFS-9254.02.patch
        39 kB
        Arpit Agarwal
      3. HDFS-9254.03.patch
        40 kB
        Arpit Agarwal
      4. HDFS-9254.04.patch
        41 kB
        Arpit Agarwal
      5. HADOOP-12570.05.patch
        89 kB
        Arpit Agarwal

        Issue Links

          Activity

          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Cherry-picked this to 2.7.3.

          Show
          arpitagarwal Arpit Agarwal added a comment - Cherry-picked this to 2.7.3.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9002 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9002/)
          HADOOP-12570. HDFS Secure Mode Documentation updates. Contributed by (cnauroth: rev 932956458dddabd9583b911c693386a56d723c0d)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsMultihoming.md
          • hadoop-common-project/hadoop-common/src/site/markdown/SecureMode.md
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-common-project/hadoop-common/src/site/markdown/HttpAuthentication.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9002 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9002/ ) HADOOP-12570 . HDFS Secure Mode Documentation updates. Contributed by (cnauroth: rev 932956458dddabd9583b911c693386a56d723c0d) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsMultihoming.md hadoop-common-project/hadoop-common/src/site/markdown/SecureMode.md hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-common-project/hadoop-common/src/site/markdown/HttpAuthentication.md
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thank you Steve and Chris.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thank you Steve and Chris.
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for patch v05. I have committed this to trunk, branch-2 and branch-2.8. Arpit, thank you for the documentation improvements. Steve, thank you for reviewing.

          Show
          cnauroth Chris Nauroth added a comment - +1 for patch v05. I have committed this to trunk, branch-2 and branch-2.8. Arpit, thank you for the documentation improvements. Steve, thank you for reviewing.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 54s trunk passed
          +1 compile 8m 5s trunk passed with JDK v1.8.0_66
          +1 compile 8m 40s trunk passed with JDK v1.7.0_91
          +1 mvnsite 1m 55s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 javadoc 2m 2s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 48s trunk passed with JDK v1.7.0_91
          +1 mvninstall 2m 33s the patch passed
          +1 compile 7m 59s the patch passed with JDK v1.8.0_66
          +1 javac 7m 59s the patch passed
          +1 compile 8m 43s the patch passed with JDK v1.7.0_91
          +1 javac 8m 43s the patch passed
          +1 mvnsite 1m 53s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 2m 1s the patch passed with JDK v1.8.0_66
          +1 javadoc 2m 51s the patch passed with JDK v1.7.0_91
          +1 unit 7m 25s hadoop-common in the patch passed with JDK v1.8.0_66.
          -1 unit 51m 16s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 7m 6s hadoop-common in the patch failed with JDK v1.7.0_91.
          -1 unit 49m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 27s Patch does not generate ASF License warnings.
          175m 30s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
          JDK v1.7.0_91 Failed junit tests hadoop.http.TestHttpServer
            hadoop.hdfs.TestRenameWhileOpen



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778392/HADOOP-12570.05.patch
          JIRA Issue HADOOP-12570
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit xml
          uname Linux d8ecbdc05c6e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6da9dee
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 54s trunk passed +1 compile 8m 5s trunk passed with JDK v1.8.0_66 +1 compile 8m 40s trunk passed with JDK v1.7.0_91 +1 mvnsite 1m 55s trunk passed +1 mvneclipse 0m 28s trunk passed +1 javadoc 2m 2s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 48s trunk passed with JDK v1.7.0_91 +1 mvninstall 2m 33s the patch passed +1 compile 7m 59s the patch passed with JDK v1.8.0_66 +1 javac 7m 59s the patch passed +1 compile 8m 43s the patch passed with JDK v1.7.0_91 +1 javac 8m 43s the patch passed +1 mvnsite 1m 53s the patch passed +1 mvneclipse 0m 28s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 2m 1s the patch passed with JDK v1.8.0_66 +1 javadoc 2m 51s the patch passed with JDK v1.7.0_91 +1 unit 7m 25s hadoop-common in the patch passed with JDK v1.8.0_66. -1 unit 51m 16s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 7m 6s hadoop-common in the patch failed with JDK v1.7.0_91. -1 unit 49m 59s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 175m 30s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits JDK v1.7.0_91 Failed junit tests hadoop.http.TestHttpServer   hadoop.hdfs.TestRenameWhileOpen Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778392/HADOOP-12570.05.patch JIRA Issue HADOOP-12570 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit xml uname Linux d8ecbdc05c6e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6da9dee whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8270/console This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          v05 patch addresses feedback from Steve:

          • Made formatting of the default values consistent. A lot of the existing settings were inconsistently formatted so fixed those too, hence the increase in patch size.

          And Chris:

          • Restored defaults to hdfs-default.xml and reverted test changes
          • Fixed broken hyperlink in HdfsMultihoming.md, added a link to the HADOOP-4487 design doc.

          Also pretty-printed the markdown tables as they were getting messy to read.

          Show
          arpitagarwal Arpit Agarwal added a comment - v05 patch addresses feedback from Steve: Made formatting of the default values consistent. A lot of the existing settings were inconsistently formatted so fixed those too, hence the increase in patch size. And Chris: Restored defaults to hdfs-default.xml and reverted test changes Fixed broken hyperlink in HdfsMultihoming.md, added a link to the HADOOP-4487 design doc. Also pretty-printed the markdown tables as they were getting messy to read.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          If an hdfs-site.xml defined dfs.web.authentication.kerberos.principal, but did not define dfs.namenode.kerberos.internal.spnego.principal, then I think it still would have worked because of the token substitution logic in Configuration#substituteVars.

          Nice catch Chris! So the reason this works is because hdfs-default.xml is available as a resource in the hadoop-hdfs jar.

          I'll update the next patch to restore the defaults.

          Show
          arpitagarwal Arpit Agarwal added a comment - If an hdfs-site.xml defined dfs.web.authentication.kerberos.principal, but did not define dfs.namenode.kerberos.internal.spnego.principal, then I think it still would have worked because of the token substitution logic in Configuration#substituteVars. Nice catch Chris! So the reason this works is because hdfs-default.xml is available as a resource in the hadoop-hdfs jar. I'll update the next patch to restore the defaults.
          Hide
          cnauroth Chris Nauroth added a comment -

          Taking this example from the existing hdfs-default.xml:

          <property>
            <name>dfs.namenode.kerberos.internal.spnego.principal</name>
            <value>${dfs.web.authentication.kerberos.principal}</value>
          </property>
          

          If an hdfs-site.xml defined dfs.web.authentication.kerberos.principal, but did not define dfs.namenode.kerberos.internal.spnego.principal, then I think it still would have worked because of the token substitution logic in Configuration#substituteVars. In this patch, dfs.namenode.kerberos.internal.spnego.principal defaults to nothing, so the token substitution won't happen anymore. This is why I think there could be a compatibility problem.

          Show
          cnauroth Chris Nauroth added a comment - Taking this example from the existing hdfs-default.xml: <property> <name>dfs.namenode.kerberos.internal.spnego.principal</name> <value>${dfs.web.authentication.kerberos.principal}</value> </property> If an hdfs-site.xml defined dfs.web.authentication.kerberos.principal , but did not define dfs.namenode.kerberos.internal.spnego.principal , then I think it still would have worked because of the token substitution logic in Configuration#substituteVars . In this patch, dfs.namenode.kerberos.internal.spnego.principal defaults to nothing, so the token substitution won't happen anymore. This is why I think there could be a compatibility problem.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Chris Nauroth, thanks for the review!

          The descriptions in hdfs-default.xml are nice to have, but I'm concerned that they are not backwards-compatible.

          There should be no incompatibility introduced by the patch. It just fixes wrong documentation to be in line with the code. e.g. dfs.namenode.kerberos.internal.spnego.principal has no default in code but it was documented as defaulting to dfs.web.authentication.kerberos.principal in hdfs-default.xml. From NameNodeHttpServer#start:

              HttpServer2.Builder builder = DFSUtil.httpServerTemplateForNNAndJN(conf,
                  httpAddr, httpsAddr, "hdfs",
                  DFSConfigKeys.DFS_NAMENODE_KERBEROS_INTERNAL_SPNEGO_PRINCIPAL_KEY,
                  DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY);
          

          A few unit tests broke after this patch because they were pulling hdfs-default.xml as a resource and depending on the incorrect defaults instead of initializing their own test configuration correctly.

          I'll post an updated patch addressing the remaining feedback.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Chris Nauroth , thanks for the review! The descriptions in hdfs-default.xml are nice to have, but I'm concerned that they are not backwards-compatible. There should be no incompatibility introduced by the patch. It just fixes wrong documentation to be in line with the code. e.g. dfs.namenode.kerberos.internal.spnego.principal has no default in code but it was documented as defaulting to dfs.web.authentication.kerberos.principal in hdfs-default.xml. From NameNodeHttpServer#start : HttpServer2.Builder builder = DFSUtil.httpServerTemplateForNNAndJN(conf, httpAddr, httpsAddr, "hdfs" , DFSConfigKeys.DFS_NAMENODE_KERBEROS_INTERNAL_SPNEGO_PRINCIPAL_KEY, DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY); A few unit tests broke after this patch because they were pulling hdfs-default.xml as a resource and depending on the incorrect defaults instead of initializing their own test configuration correctly. I'll post an updated patch addressing the remaining feedback.
          Hide
          cnauroth Chris Nauroth added a comment -

          Nice improvements. Thank you, Arpit.

          In the references, the first entry can link to HADOOP-4487, which is the original Hadoop security JIRA with design doc attached.

          In HdfsMultiHoming.md, the "Hostname Lookup" link isn't rendering correctly. There is a missing underscore. This would fix it:

                  * [Hostname Lookup](#Hostname_Lookup)
          

          The descriptions in hdfs-default.xml are nice to have, but I'm concerned that they are not backwards-compatible. dfs.namenode.kerberos.internal.spnego.principal used to default to using the value of dfs.web.authentication.kerberos.principal. That would mean that if someone had configured dfs.web.authentication.kerberos.principal, but not dfs.namenode.kerberos.internal.spnego.principal, then it could have worked before this patch, but it would stop working after this patch.

          That probably partly explains the earlier SaslDataTransferTestCase failures. I think if tests need to be changed for this patch, then it's probably a sign that something has gone wrong.

          Show
          cnauroth Chris Nauroth added a comment - Nice improvements. Thank you, Arpit. In the references, the first entry can link to HADOOP-4487 , which is the original Hadoop security JIRA with design doc attached. In HdfsMultiHoming.md, the "Hostname Lookup" link isn't rendering correctly. There is a missing underscore. This would fix it: * [Hostname Lookup](#Hostname_Lookup) The descriptions in hdfs-default.xml are nice to have, but I'm concerned that they are not backwards-compatible. dfs.namenode.kerberos.internal.spnego.principal used to default to using the value of dfs.web.authentication.kerberos.principal . That would mean that if someone had configured dfs.web.authentication.kerberos.principal , but not dfs.namenode.kerberos.internal.spnego.principal , then it could have worked before this patch, but it would stop working after this patch. That probably partly explains the earlier SaslDataTransferTestCase failures. I think if tests need to be changed for this patch, then it's probably a sign that something has gone wrong.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Some of the default values in tables are emphasised; others converted to `code` font. I think they should all be `in code`.

          You can do `italiccode` for both

          line 271,

          /etc/security/keytab/nn.service.keytab should be ` /etc/security/keytab/nn.service.keytab`

          same for line 273 and /etc/security/keytab/spnego.service.keytab

          ... etc

          otherwise, +1

          Show
          stevel@apache.org Steve Loughran added a comment - Some of the default values in tables are emphasised ; others converted to `code` font. I think they should all be `in code`. You can do `italiccode` for both line 271, /etc/security/keytab/nn.service.keytab should be ` /etc/security/keytab/nn.service.keytab` same for line 273 and /etc/security/keytab/spnego.service.keytab ... etc otherwise, +1
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          I have no idea what that's about. So I am going to see if I can find another reviewer.

          Show
          arpitagarwal Arpit Agarwal added a comment - I have no idea what that's about. So I am going to see if I can find another reviewer.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          yeah, find a way to make my ZK client code not to leave GSS API error 44 in the zk logs

          Show
          stevel@apache.org Steve Loughran added a comment - yeah, find a way to make my ZK client code not to leave GSS API error 44 in the zk logs
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Steve Loughran, any way I can convince you to review the latest patch? I think it addresses your feedback.

          Show
          arpitagarwal Arpit Agarwal added a comment - Steve Loughran , any way I can convince you to review the latest patch? I think it addresses your feedback.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          The failures look unrelated to the patch.

          Show
          arpitagarwal Arpit Agarwal added a comment - The failures look unrelated to the patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 6s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 2m 59s trunk passed
          +1 compile 4m 23s trunk passed with JDK v1.8.0_60
          +1 compile 4m 15s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 57s trunk passed
          +1 mvnsite 1m 36s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 3m 33s trunk passed
          +1 javadoc 2m 2s trunk passed with JDK v1.8.0_60
          +1 javadoc 2m 50s trunk passed with JDK v1.7.0_79
          +1 mvninstall 2m 12s the patch passed
          +1 compile 4m 26s the patch passed with JDK v1.8.0_60
          +1 javac 4m 26s the patch passed
          +1 compile 4m 12s the patch passed with JDK v1.7.0_79
          +1 javac 4m 12s the patch passed
          +1 checkstyle 0m 56s the patch passed
          +1 mvnsite 1m 36s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 49s the patch passed
          +1 javadoc 2m 1s the patch passed with JDK v1.8.0_60
          +1 javadoc 2m 54s the patch passed with JDK v1.7.0_79
          -1 unit 16m 58s hadoop-common in the patch failed with JDK v1.8.0_60.
          -1 unit 51m 47s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          -1 unit 7m 40s hadoop-common in the patch failed with JDK v1.7.0_79.
          -1 unit 52m 53s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 20s Patch generated 56 ASF License warnings.
          176m 39s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.hdfs.server.namenode.ha.TestDNFencing
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          JDK v1.8.0_60 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
          JDK v1.7.0_79 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.fs.TestLocalFsFCStatistics
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-14
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772315/HDFS-9254.04.patch
          JIRA Issue HADOOP-12570
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit xml findbugs checkstyle
          uname Linux 42053c7cda06 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh
          git revision trunk / f94d892
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 225MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 2m 59s trunk passed +1 compile 4m 23s trunk passed with JDK v1.8.0_60 +1 compile 4m 15s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 57s trunk passed +1 mvnsite 1m 36s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 3m 33s trunk passed +1 javadoc 2m 2s trunk passed with JDK v1.8.0_60 +1 javadoc 2m 50s trunk passed with JDK v1.7.0_79 +1 mvninstall 2m 12s the patch passed +1 compile 4m 26s the patch passed with JDK v1.8.0_60 +1 javac 4m 26s the patch passed +1 compile 4m 12s the patch passed with JDK v1.7.0_79 +1 javac 4m 12s the patch passed +1 checkstyle 0m 56s the patch passed +1 mvnsite 1m 36s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 49s the patch passed +1 javadoc 2m 1s the patch passed with JDK v1.8.0_60 +1 javadoc 2m 54s the patch passed with JDK v1.7.0_79 -1 unit 16m 58s hadoop-common in the patch failed with JDK v1.8.0_60. -1 unit 51m 47s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 7m 40s hadoop-common in the patch failed with JDK v1.7.0_79. -1 unit 52m 53s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 20s Patch generated 56 ASF License warnings. 176m 39s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes JDK v1.8.0_60 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle JDK v1.7.0_79 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.fs.TestLocalFsFCStatistics   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.ha.TestEditLogTailer Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-14 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772315/HDFS-9254.04.patch JIRA Issue HADOOP-12570 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit xml findbugs checkstyle uname Linux 42053c7cda06 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh git revision trunk / f94d892 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 225MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8070/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s docker + precommit patch detected.
          +1 @author 0m 1s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 3m 15s trunk passed
          +1 compile 5m 19s trunk passed with JDK v1.8.0_66
          +1 compile 5m 33s trunk passed with JDK v1.7.0_79
          +1 checkstyle 1m 3s trunk passed
          +1 mvnsite 1m 58s trunk passed
          +1 mvneclipse 0m 46s trunk passed
          +1 findbugs 5m 4s trunk passed
          +1 javadoc 2m 52s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 20s trunk passed with JDK v1.7.0_79
          +1 mvninstall 2m 23s the patch passed
          +1 compile 5m 14s the patch passed with JDK v1.8.0_66
          +1 javac 5m 14s the patch passed
          +1 compile 4m 55s the patch passed with JDK v1.7.0_79
          +1 javac 4m 55s the patch passed
          +1 checkstyle 1m 6s the patch passed
          +1 mvnsite 1m 51s the patch passed
          +1 mvneclipse 0m 32s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 4m 35s the patch passed
          +1 javadoc 2m 30s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 19s the patch passed with JDK v1.7.0_79
          -1 unit 8m 44s hadoop-common in the patch failed with JDK v1.8.0_66.
          -1 unit 78m 53s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 9m 13s hadoop-common in the patch passed with JDK v1.7.0_79.
          -1 unit 71m 3s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 22s Patch generated 56 ASF License warnings.
          225m 29s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestMetricsSystemImpl
            hadoop.ipc.TestRPCWaitForProxy
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
            hadoop.hdfs.TestDFSClientRetries



          Subsystem Report/Notes
          Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-13
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772315/HDFS-9254.04.patch
          JIRA Issue HDFS-9254
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit xml findbugs checkstyle
          uname Linux 552f8f9cbc2b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh
          git revision trunk / f94d892
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13507/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 227MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13507/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s docker + precommit patch detected. +1 @author 0m 1s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 3m 15s trunk passed +1 compile 5m 19s trunk passed with JDK v1.8.0_66 +1 compile 5m 33s trunk passed with JDK v1.7.0_79 +1 checkstyle 1m 3s trunk passed +1 mvnsite 1m 58s trunk passed +1 mvneclipse 0m 46s trunk passed +1 findbugs 5m 4s trunk passed +1 javadoc 2m 52s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 20s trunk passed with JDK v1.7.0_79 +1 mvninstall 2m 23s the patch passed +1 compile 5m 14s the patch passed with JDK v1.8.0_66 +1 javac 5m 14s the patch passed +1 compile 4m 55s the patch passed with JDK v1.7.0_79 +1 javac 4m 55s the patch passed +1 checkstyle 1m 6s the patch passed +1 mvnsite 1m 51s the patch passed +1 mvneclipse 0m 32s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 35s the patch passed +1 javadoc 2m 30s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 19s the patch passed with JDK v1.7.0_79 -1 unit 8m 44s hadoop-common in the patch failed with JDK v1.8.0_66. -1 unit 78m 53s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 9m 13s hadoop-common in the patch passed with JDK v1.7.0_79. -1 unit 71m 3s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 22s Patch generated 56 ASF License warnings. 225m 29s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestMetricsSystemImpl   hadoop.ipc.TestRPCWaitForProxy   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestDFSClientRetries Subsystem Report/Notes Docker Client=1.7.0 Server=1.7.0 Image:test-patch-base-hadoop-date2015-11-13 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772315/HDFS-9254.04.patch JIRA Issue HDFS-9254 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit xml findbugs checkstyle uname Linux 552f8f9cbc2b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh git revision trunk / f94d892 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13507/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13507/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 227MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13507/console This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for the review. I've added a references section including your book and fixed the tabs issue in hdfs-default.xml.

          I referred to your book when I was stuck debugging SPNEGO auth issues recently but didn't find an answer that helped me. Finally tracked it down to DNS-related quirks. It prompted me to document the parts I understood but I still don't understand SPNEGO well enough to improve its documentation. I don't envy the next person who has to debug Hadoop security quirks.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for the review. I've added a references section including your book and fixed the tabs issue in hdfs-default.xml. I referred to your book when I was stuck debugging SPNEGO auth issues recently but didn't find an answer that helped me. Finally tracked it down to DNS-related quirks. It prompted me to document the parts I understood but I still don't understand SPNEGO well enough to improve its documentation. I don't envy the next person who has to debug Hadoop security quirks.
          Hide
          stevel@apache.org Steve Loughran added a comment -
          1. check the indentation on some of the XML examples
          2. doesn't my little ebook merit a reference?
          3. you can paste in it's references too, or at least a subset of them: https://github.com/steveloughran/kerberos_and_hadoop/blob/master/sections/biblography.md
          Show
          stevel@apache.org Steve Loughran added a comment - check the indentation on some of the XML examples doesn't my little ebook merit a reference? you can paste in it's references too, or at least a subset of them: https://github.com/steveloughran/kerberos_and_hadoop/blob/master/sections/biblography.md
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 6s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 3m 13s trunk passed
          +1 compile 4m 23s trunk passed with JDK v1.8.0_60
          +1 compile 4m 6s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 55s trunk passed
          +1 mvnsite 1m 34s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          -1 findbugs 1m 51s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
          +1 javadoc 2m 0s trunk passed with JDK v1.8.0_60
          +1 javadoc 2m 53s trunk passed with JDK v1.7.0_79
          +1 mvninstall 2m 15s the patch passed
          +1 compile 4m 20s the patch passed with JDK v1.8.0_60
          +1 javac 4m 20s the patch passed
          +1 compile 4m 17s the patch passed with JDK v1.7.0_79
          +1 javac 4m 17s the patch passed
          +1 checkstyle 0m 59s the patch passed
          +1 mvnsite 1m 32s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          -1 whitespace 0m 0s The patch has 34 line(s) with tabs.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 3m 48s the patch passed
          +1 javadoc 1m 58s the patch passed with JDK v1.8.0_60
          +1 javadoc 2m 48s the patch passed with JDK v1.7.0_79
          -1 unit 6m 4s hadoop-common in the patch failed with JDK v1.8.0_60.
          -1 unit 62m 21s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
          +1 unit 6m 49s hadoop-common in the patch passed with JDK v1.7.0_79.
          -1 unit 59m 4s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
          -1 asflicense 0m 24s Patch generated 58 ASF License warnings.
          181m 30s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.security.TestShellBasedIdMapping
            hadoop.hdfs.server.namenode.TestRecoverStripedBlocks
            hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
            hadoop.hdfs.TestDFSStripedOutputStream
            hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
            hadoop.hdfs.TestReplication
            hadoop.hdfs.server.namenode.TestAddStripedBlocks
            hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
            hadoop.hdfs.TestSafeModeWithStripedFile
            hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
            hadoop.hdfs.server.namenode.TestAddBlockRetry
            hadoop.hdfs.server.namenode.TestAddBlock
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup
            hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170
            hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
            hadoop.hdfs.server.mover.TestMover
            hadoop.hdfs.TestFileCreation
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
            hadoop.hdfs.server.datanode.TestHSync
            hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
            hadoop.hdfs.server.namenode.TestFsck
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.server.blockmanagement.TestPendingReplication
            hadoop.fs.permission.TestStickyBit
            hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
            hadoop.hdfs.TestBlockStoragePolicy
            hadoop.hdfs.TestGetFileChecksum
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
            hadoop.hdfs.TestDecommission
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
            hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant
            hadoop.hdfs.server.datanode.TestDiskError
            hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.hdfs.TestRecoverStripedFile
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
            hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
            hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
            hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.TestReadWhileWriting
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120
            hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
            hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
            hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
            hadoop.hdfs.TestWriteReadStripedFile
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
            hadoop.hdfs.TestQuota
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000
            hadoop.hdfs.TestInjectionForSimulatedStorage
          JDK v1.8.0_60 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
          JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.TestRecoverStripedBlocks
            hadoop.hdfs.server.datanode.TestDataNodeMetrics
            hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
            hadoop.hdfs.TestDFSStripedOutputStream
            hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
            hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages
            hadoop.hdfs.server.namenode.TestAddStripedBlocks
            hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
            hadoop.hdfs.TestSafeModeWithStripedFile
            hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
            hadoop.hdfs.server.namenode.TestAddBlockRetry
            hadoop.hdfs.server.namenode.ha.TestStandbyIsHot
            hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup
            hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
            hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
            hadoop.hdfs.server.namenode.TestSnapshotPathINodes
            hadoop.hdfs.server.datanode.TestDeleteBlockPool
            hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
            hadoop.hdfs.server.mover.TestMover
            hadoop.hdfs.TestFileCreation
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210
            hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
            hadoop.hdfs.server.datanode.TestHSync
            hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
            hadoop.hdfs.server.namenode.TestFsck
            hadoop.hdfs.server.blockmanagement.TestNodeCount
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.server.blockmanagement.TestPendingReplication
            hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
            hadoop.hdfs.TestBlockStoragePolicy
            hadoop.hdfs.TestGetFileChecksum
            hadoop.hdfs.TestReadStripedFileWithMissingBlocks
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
            hadoop.hdfs.TestDecommission
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
            hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant
            hadoop.hdfs.server.datanode.TestDiskError
            hadoop.hdfs.TestReadStripedFileWithDecoding
            hadoop.hdfs.TestRecoverStripedFile
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.namenode.TestMetaSave
            hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
            hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
            hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
            hadoop.hdfs.TestEncryptedTransfer
            hadoop.hdfs.server.blockmanagement.TestBlockManager
            hadoop.hdfs.server.namenode.ha.TestDNFencing
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy
            hadoop.hdfs.TestReplaceDatanodeOnFailure
            hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
            hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
            hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
            hadoop.hdfs.TestWriteReadStripedFile
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000
            hadoop.hdfs.TestInjectionForSimulatedStorage
          JDK v1.7.0_79 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-31
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769874/HDFS-9254.03.patch
          JIRA Issue HDFS-9254
          Optional Tests asflicense site mvnsite javac javadoc mvninstall unit xml findbugs checkstyle compile
          uname Linux eb9252cb1415 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh
          git revision trunk / b24fe06
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13309/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Max memory used 226MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13309/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 6s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 3m 13s trunk passed +1 compile 4m 23s trunk passed with JDK v1.8.0_60 +1 compile 4m 6s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 55s trunk passed +1 mvnsite 1m 34s trunk passed +1 mvneclipse 0m 28s trunk passed -1 findbugs 1m 51s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 2m 0s trunk passed with JDK v1.8.0_60 +1 javadoc 2m 53s trunk passed with JDK v1.7.0_79 +1 mvninstall 2m 15s the patch passed +1 compile 4m 20s the patch passed with JDK v1.8.0_60 +1 javac 4m 20s the patch passed +1 compile 4m 17s the patch passed with JDK v1.7.0_79 +1 javac 4m 17s the patch passed +1 checkstyle 0m 59s the patch passed +1 mvnsite 1m 32s the patch passed +1 mvneclipse 0m 27s the patch passed -1 whitespace 0m 0s The patch has 34 line(s) with tabs. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 3m 48s the patch passed +1 javadoc 1m 58s the patch passed with JDK v1.8.0_60 +1 javadoc 2m 48s the patch passed with JDK v1.7.0_79 -1 unit 6m 4s hadoop-common in the patch failed with JDK v1.8.0_60. -1 unit 62m 21s hadoop-hdfs in the patch failed with JDK v1.8.0_60. +1 unit 6m 49s hadoop-common in the patch passed with JDK v1.7.0_79. -1 unit 59m 4s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 24s Patch generated 58 ASF License warnings. 181m 30s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.security.TestShellBasedIdMapping   hadoop.hdfs.server.namenode.TestRecoverStripedBlocks   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStream   hadoop.hdfs.server.datanode.TestReadOnlySharedStorage   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestReplication   hadoop.hdfs.server.namenode.TestAddStripedBlocks   hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes   hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage   hadoop.hdfs.server.namenode.TestAddBlockRetry   hadoop.hdfs.server.namenode.TestAddBlock   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks   hadoop.hdfs.server.mover.TestMover   hadoop.hdfs.TestFileCreation   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.server.datanode.TestHSync   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.blockmanagement.TestPendingReplication   hadoop.fs.permission.TestStickyBit   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy   hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.TestGetFileChecksum   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.TestDecommission   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant   hadoop.hdfs.server.datanode.TestDiskError   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestReadWhileWriting   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot   hadoop.hdfs.TestQuota   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000   hadoop.hdfs.TestInjectionForSimulatedStorage JDK v1.8.0_60 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped JDK v1.7.0_79 Failed junit tests hadoop.hdfs.server.namenode.TestRecoverStripedBlocks   hadoop.hdfs.server.datanode.TestDataNodeMetrics   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStream   hadoop.hdfs.server.datanode.TestReadOnlySharedStorage   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages   hadoop.hdfs.server.namenode.TestAddStripedBlocks   hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes   hadoop.hdfs.TestSafeModeWithStripedFile   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage   hadoop.hdfs.server.namenode.TestAddBlockRetry   hadoop.hdfs.server.namenode.ha.TestStandbyIsHot   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.server.namenode.TestSnapshotPathINodes   hadoop.hdfs.server.datanode.TestDeleteBlockPool   hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks   hadoop.hdfs.server.mover.TestMover   hadoop.hdfs.TestFileCreation   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.server.datanode.TestHSync   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.blockmanagement.TestPendingReplication   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy   hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.TestGetFileChecksum   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.TestDecommission   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant   hadoop.hdfs.server.datanode.TestDiskError   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.TestRecoverStripedFile   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.TestMetaSave   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.namenode.TestFSImageWithSnapshot   hadoop.hdfs.TestEncryptedTransfer   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.hdfs.TestWriteReadStripedFile   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000   hadoop.hdfs.TestInjectionForSimulatedStorage JDK v1.7.0_79 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-31 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769874/HDFS-9254.03.patch JIRA Issue HDFS-9254 Optional Tests asflicense site mvnsite javac javadoc mvninstall unit xml findbugs checkstyle compile uname Linux eb9252cb1415 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh git revision trunk / b24fe06 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13309/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13309/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13309/console This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          v3 patch.

          1. I had fixed incorrect defaults in hdfs-defaults.xml which broke some secure mode tests. Fixed those tests to initialize their configuration correctly.
          2. Remove mention of ServiceName@REALM.TLD principals.
          Show
          arpitagarwal Arpit Agarwal added a comment - v3 patch. I had fixed incorrect defaults in hdfs-defaults.xml which broke some secure mode tests. Fixed those tests to initialize their configuration correctly. Remove mention of ServiceName@REALM.TLD principals.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          So yes it looks like at least the SaslRpcClient doesn't like principals without a host component.

          192.168.56.80:8485: Failed on local exception: java.io.IOException: java.lang.IllegalArgumentException: Kerberos principal name does NOT have the expected hostname part: jn@EXAMPLE.COM; Host Details : local host is: "cm0.example.com/192.168.56.80"; destination host is: "cm0.example.com":8485;
                  at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
                  at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223)
                  at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.hasSomeData(QuorumJournalManager.java:232)
                  at org.apache.hadoop.hdfs.server.common.Storage.confirmFormat(Storage.java:899)
          

          Whereas SecurityUtil handles them fine. We should be consistent. I'll file a separate bug to fix the SaslRpcClient, and any other components I run into, but also update the doc patch for now. Thanks for the catch.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited So yes it looks like at least the SaslRpcClient doesn't like principals without a host component. 192.168.56.80:8485: Failed on local exception: java.io.IOException: java.lang.IllegalArgumentException: Kerberos principal name does NOT have the expected hostname part: jn@EXAMPLE.COM; Host Details : local host is: "cm0.example.com/192.168.56.80" ; destination host is: "cm0.example.com" :8485; at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81) at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223) at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.hasSomeData(QuorumJournalManager.java:232) at org.apache.hadoop.hdfs.server.common.Storage.confirmFormat(Storage.java:899) Whereas SecurityUtil handles them fine. We should be consistent. I'll file a separate bug to fix the SaslRpcClient , and any other components I run into, but also update the doc patch for now. Thanks for the catch.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          The test failures do seem to be caused by my patch, oddly. I'll take a look.

          in the patch, you say that `dn@REALM.TLD` is allowed. I recall seeing some JIRAs where people were saying you get a stack trace unless you have the /HOST value of some kind or other

          Let me verify that in a test cluster, thanks for looking at the patch.

          Show
          arpitagarwal Arpit Agarwal added a comment - The test failures do seem to be caused by my patch, oddly. I'll take a look. in the patch, you say that `dn@REALM.TLD` is allowed. I recall seeing some JIRAs where people were saying you get a stack trace unless you have the /HOST value of some kind or other Let me verify that in a test cluster, thanks for looking at the patch.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I'd thought "underdocumented" is a complete summary of kerberos info —good to see you trying to fix this.

          in the patch, you say that `dn@REALM.TLD` is allowed. I recall seeing some JIRAs where people were saying you get a stack trace unless you have the /HOST value of some kind or other

          Show
          stevel@apache.org Steve Loughran added a comment - I'd thought "underdocumented" is a complete summary of kerberos info —good to see you trying to fix this. in the patch, you say that `dn@REALM.TLD` is allowed. I recall seeing some JIRAs where people were saying you get a stack trace unless you have the /HOST value of some kind or other
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Documentation-only patch, needs no new tests. Test failures are unrelated to the patch.

          Show
          arpitagarwal Arpit Agarwal added a comment - Documentation-only patch, needs no new tests. Test failures are unrelated to the patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 22s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 8m 1s There were no new javac warning messages.
          +1 javadoc 10m 39s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 site 3m 32s Site still builds.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 44s mvn install still works.
          +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
          +1 common tests 7m 27s Tests passed in hadoop-common.
          -1 hdfs tests 50m 51s Tests failed in hadoop-hdfs.
              102m 41s  



          Reason Tests
          Failed unit tests hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.namenode.TestSecureNameNode
            hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12766986/HDFS-9254.01.patch
          Optional Tests site javadoc javac unit
          git revision trunk / cf23f2c
          hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13025/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13025/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13025/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13025/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 22s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 1s There were no new javac warning messages. +1 javadoc 10m 39s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 site 3m 32s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 44s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 common tests 7m 27s Tests passed in hadoop-common. -1 hdfs tests 50m 51s Tests failed in hadoop-hdfs.     102m 41s   Reason Tests Failed unit tests hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.namenode.TestSecureNameNode   hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766986/HDFS-9254.01.patch Optional Tests site javadoc javac unit git revision trunk / cf23f2c hadoop-common test log https://builds.apache.org/job/PreCommit-HDFS-Build/13025/artifact/patchprocess/testrun_hadoop-common.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/13025/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13025/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13025/console This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          v1 patch.

          1. Documents missing settings in hdfs-default.xml.
          2. Add more detail to SecureMode.html, rewrite some sections. Document JournalNode settings.
          3. Update HdfsMultihoming.html to document new security settings added by HADOOP-12437.
          4. Minor cleanup of HttpAuthentication.html to convert textual description to a table. I think this doc needs more detail but I don't understand this part of the configuration well enough to add content.
          Show
          arpitagarwal Arpit Agarwal added a comment - v1 patch. Documents missing settings in hdfs-default.xml. Add more detail to SecureMode.html, rewrite some sections. Document JournalNode settings. Update HdfsMultihoming.html to document new security settings added by HADOOP-12437 . Minor cleanup of HttpAuthentication.html to convert textual description to a table. I think this doc needs more detail but I don't understand this part of the configuration well enough to add content.

            People

            • Assignee:
              arpitagarwal Arpit Agarwal
              Reporter:
              arpitagarwal Arpit Agarwal
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development