Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: security
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hadoop now includes a shell command named KDiag that helps with diagnosis of Kerberos misconfiguration problems. Please refer to the Secure Mode documentation for full details on usage of the command.

      Description

      If we a little command line entry point for testing kerberos settings, including some automated diagnostics checks, we could simplify fielding the client-side support calls.

      Specifically

      • check JRE for having java crypto extensions at full key length.
      • network checks: do you know your own name?
      • Is the user kinited in?
      • if a tgt is specified, does it exist?
      • are hadoop security options consistent?
      1. HADOOP-12426-010.patch
        57 kB
        Steve Loughran
      2. HADOOP-12426-009.patch
        57 kB
        Steve Loughran
      3. HADOOP-12426-008.patch
        48 kB
        Steve Loughran
      4. HADOOP-12426-007.patch
        47 kB
        Steve Loughran
      5. HADOOP-12426-006.patch
        37 kB
        Steve Loughran
      6. HADOOP-12426-004.patch
        29 kB
        Steve Loughran
      7. HADOOP-12426-003.patch
        29 kB
        Steve Loughran
      8. HADOOP-12426-002.patch
        28 kB
        Steve Loughran
      9. HADOOP-12426-001.patch
        28 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          drankye Kai Zheng added a comment -

          Thanks Steve.

          This looks cool. One thing in mind, are you expecting this client-side tool to be runnable on NN and DNs side? What behavior would be if it's runned there? Wondering if it would be good to consider service keytab check.

          I guess the new funcationalities will be added to 'hadoop' command, implemented in Java codes, right.

          Show
          drankye Kai Zheng added a comment - Thanks Steve. This looks cool. One thing in mind, are you expecting this client-side tool to be runnable on NN and DNs side? What behavior would be if it's runned there? Wondering if it would be good to consider service keytab check. I guess the new funcationalities will be added to 'hadoop' command, implemented in Java codes, right.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I'm just looking for some entry point that can be run in bin/hadoop to check keytab health before you get odd errors; something vaguely useful for end users, and at least for people fielding support calls.

          bin/hadoop kerberos-check --keytab ~/keys --principal stevel@LOCAL
          

          this could trigger

          1. verify keytab is there, readable, contains the named principal
          2. can actually generate keys the right length with JCE
          3. KDC is resolvable
          4. KDC exists (outages may be transient, but stil...)

          and there'd be some different negative exit codes for failures.

          I don't have the time to do this right now, I just think it'd be good for preflight checking. If the class to run the tests was also made usable from as a library, I think checking JCE key length and principal would be useful too -especially if it could generate some more useful message than those we see today.

          Show
          stevel@apache.org Steve Loughran added a comment - I'm just looking for some entry point that can be run in bin/hadoop to check keytab health before you get odd errors; something vaguely useful for end users, and at least for people fielding support calls. bin/hadoop kerberos-check --keytab ~/keys --principal stevel@LOCAL this could trigger verify keytab is there, readable, contains the named principal can actually generate keys the right length with JCE KDC is resolvable KDC exists (outages may be transient, but stil...) and there'd be some different negative exit codes for failures. I don't have the time to do this right now, I just think it'd be good for preflight checking. If the class to run the tests was also made usable from as a library, I think checking JCE key length and principal would be useful too -especially if it could generate some more useful message than those we see today.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          The UGI improvements of HADOOP-12649 are a foundation for this

          Show
          stevel@apache.org Steve Loughran added a comment - The UGI improvements of HADOOP-12649 are a foundation for this
          Hide
          stevel@apache.org Steve Loughran added a comment -

          SLIDER-1027 is the first implementation of this; it's in the hadoop security package and only using hadoop operations as a precursor to migration. I just need it without waiting for a future hadoop release.

          Show
          stevel@apache.org Steve Loughran added a comment - SLIDER-1027 is the first implementation of this; it's in the hadoop security package and only using hadoop operations as a precursor to migration. I just need it without waiting for a future hadoop release.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch -001

          this is derived from the slider kdiag, adding a toolrunner entry point, some more checks, and some basic tests.

          Even with a UGI.reset() called between test methods, the JVM state is a mess: if you try to log in as an unknown principal, you are still assumed to be that principal. Accordingly: one test is commented out.

          Test-wise, this is the limit of what you can do in unit tests.

          I've not added this as an end-user command, not yet...it needs playing with.

          Usage

          kdiag [--keytab <keytab> --principal <principal>] [--out <outfile>] [--keylength <length>] [--secure]
          

          the -secure flag fails fast on an insecure cluster

          Show
          stevel@apache.org Steve Loughran added a comment - Patch -001 this is derived from the slider kdiag, adding a toolrunner entry point, some more checks, and some basic tests. Even with a UGI.reset() called between test methods, the JVM state is a mess: if you try to log in as an unknown principal, you are still assumed to be that principal. Accordingly: one test is commented out. Test-wise, this is the limit of what you can do in unit tests. I've not added this as an end-user command, not yet...it needs playing with. Usage kdiag [--keytab <keytab> --principal <principal>] [--out <outfile>] [--keylength <length>] [--secure] the -secure flag fails fast on an insecure cluster
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +we'd all benefit from the initial code in 2.8; then stabilise it. this tool matters: I've already been recommending it to people.

          Show
          stevel@apache.org Steve Loughran added a comment - +we'd all benefit from the initial code in 2.8; then stabilise it. this tool matters: I've already been recommending it to people.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 1s trunk passed
          +1 compile 6m 57s trunk passed with JDK v1.8.0_66
          +1 compile 7m 36s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 1m 6s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 59s trunk passed
          +1 javadoc 0m 58s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 9s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 39s the patch passed
          +1 compile 6m 58s the patch passed with JDK v1.8.0_66
          +1 javac 6m 58s the patch passed
          +1 compile 7m 34s the patch passed with JDK v1.7.0_91
          +1 javac 7m 34s the patch passed
          -1 checkstyle 0m 17s hadoop-common-project/hadoop-common: patch generated 71 new + 0 unchanged - 0 fixed = 71 total (was 0)
          +1 mvnsite 1m 5s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 2m 18s hadoop-common-project/hadoop-common generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javadoc 0m 57s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 10s the patch passed with JDK v1.7.0_91
          -1 unit 7m 48s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 8m 19s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          68m 16s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            Boxing/unboxing to parse a primitive org.apache.hadoop.security.KDiag.run(String[]) At KDiag.java:org.apache.hadoop.security.KDiag.run(String[]) At KDiag.java:[line 146]
            Found reliance on default encoding in org.apache.hadoop.security.KDiag.run(String[]):in org.apache.hadoop.security.KDiag.run(String[]): new java.io.PrintWriter(OutputStream) At KDiag.java:[line 142]
          JDK v1.8.0_66 Failed junit tests hadoop.ipc.TestIPC



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782822/HADOOP-12426-001.patch
          JIRA Issue HADOOP-12426
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8a3e9fdc2c7c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b08ecf5
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8434/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/8434/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8434/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8434/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8434/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8434/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 1s trunk passed +1 compile 6m 57s trunk passed with JDK v1.8.0_66 +1 compile 7m 36s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 17s trunk passed +1 mvnsite 1m 6s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 59s trunk passed +1 javadoc 0m 58s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 9s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 39s the patch passed +1 compile 6m 58s the patch passed with JDK v1.8.0_66 +1 javac 6m 58s the patch passed +1 compile 7m 34s the patch passed with JDK v1.7.0_91 +1 javac 7m 34s the patch passed -1 checkstyle 0m 17s hadoop-common-project/hadoop-common: patch generated 71 new + 0 unchanged - 0 fixed = 71 total (was 0) +1 mvnsite 1m 5s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 2m 18s hadoop-common-project/hadoop-common generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 0m 57s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 10s the patch passed with JDK v1.7.0_91 -1 unit 7m 48s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 8m 19s hadoop-common in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 68m 16s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   Boxing/unboxing to parse a primitive org.apache.hadoop.security.KDiag.run(String[]) At KDiag.java:org.apache.hadoop.security.KDiag.run(String[]) At KDiag.java: [line 146]   Found reliance on default encoding in org.apache.hadoop.security.KDiag.run(String[]):in org.apache.hadoop.security.KDiag.run(String[]): new java.io.PrintWriter(OutputStream) At KDiag.java: [line 142] JDK v1.8.0_66 Failed junit tests hadoop.ipc.TestIPC Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782822/HADOOP-12426-001.patch JIRA Issue HADOOP-12426 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8a3e9fdc2c7c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b08ecf5 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8434/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/8434/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8434/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8434/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8434/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8434/console This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch -002

          1. addresses findbug issues
          2. addresses checkstyle issues related to line endings, encoding, whitespace in statements
          3. pointedly refuses to comply with checkstyle's arbitrary policy about what constitutes valid indentation, on the basis that it is attempting to impose rules which are not in the sun guidelines.
          Show
          stevel@apache.org Steve Loughran added a comment - Patch -002 addresses findbug issues addresses checkstyle issues related to line endings, encoding, whitespace in statements pointedly refuses to comply with checkstyle's arbitrary policy about what constitutes valid indentation, on the basis that it is attempting to impose rules which are not in the sun guidelines.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 6s trunk passed
          +1 compile 7m 0s trunk passed with JDK v1.8.0_66
          +1 compile 7m 35s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 16s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 58s trunk passed
          +1 javadoc 0m 58s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 9s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 38s the patch passed
          +1 compile 7m 3s the patch passed with JDK v1.8.0_66
          +1 javac 7m 3s the patch passed
          +1 compile 7m 38s the patch passed with JDK v1.7.0_91
          +1 javac 7m 38s the patch passed
          -1 checkstyle 0m 18s hadoop-common-project/hadoop-common: patch generated 71 new + 0 unchanged - 0 fixed = 71 total (was 0)
          +1 mvnsite 1m 6s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 2m 16s hadoop-common-project/hadoop-common generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javadoc 0m 56s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 7s the patch passed with JDK v1.7.0_91
          +1 unit 8m 28s hadoop-common in the patch passed with JDK v1.8.0_66.
          +1 unit 8m 22s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          69m 13s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            Boxing/unboxing to parse a primitive org.apache.hadoop.security.KDiag.run(String[]) At KDiag.java:org.apache.hadoop.security.KDiag.run(String[]) At KDiag.java:[line 146]
            Found reliance on default encoding in org.apache.hadoop.security.KDiag.run(String[]):in org.apache.hadoop.security.KDiag.run(String[]): new java.io.PrintWriter(OutputStream) At KDiag.java:[line 142]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782924/HADOOP-12426-002.patch
          JIRA Issue HADOOP-12426
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b3db7d12cfaa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8bc93db
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8435/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/8435/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8435/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8435/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 6s trunk passed +1 compile 7m 0s trunk passed with JDK v1.8.0_66 +1 compile 7m 35s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 16s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 0m 58s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 9s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 38s the patch passed +1 compile 7m 3s the patch passed with JDK v1.8.0_66 +1 javac 7m 3s the patch passed +1 compile 7m 38s the patch passed with JDK v1.7.0_91 +1 javac 7m 38s the patch passed -1 checkstyle 0m 18s hadoop-common-project/hadoop-common: patch generated 71 new + 0 unchanged - 0 fixed = 71 total (was 0) +1 mvnsite 1m 6s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 2m 16s hadoop-common-project/hadoop-common generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 0m 56s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 7s the patch passed with JDK v1.7.0_91 +1 unit 8m 28s hadoop-common in the patch passed with JDK v1.8.0_66. +1 unit 8m 22s hadoop-common in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 69m 13s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   Boxing/unboxing to parse a primitive org.apache.hadoop.security.KDiag.run(String[]) At KDiag.java:org.apache.hadoop.security.KDiag.run(String[]) At KDiag.java: [line 146]   Found reliance on default encoding in org.apache.hadoop.security.KDiag.run(String[]):in org.apache.hadoop.security.KDiag.run(String[]): new java.io.PrintWriter(OutputStream) At KDiag.java: [line 142] Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782924/HADOOP-12426-002.patch JIRA Issue HADOOP-12426 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b3db7d12cfaa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8bc93db Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8435/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/8435/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8435/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8435/console This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          patch 002 didn't have the updated diffs in

          Show
          stevel@apache.org Steve Loughran added a comment - patch 002 didn't have the updated diffs in
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 003, has the improvements promised in 002

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 003, has the improvements promised in 002
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 6s trunk passed
          +1 compile 6m 59s trunk passed with JDK v1.8.0_66
          +1 compile 7m 41s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 1m 7s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 59s trunk passed
          +1 javadoc 0m 59s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 9s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 38s the patch passed
          +1 compile 6m 59s the patch passed with JDK v1.8.0_66
          +1 javac 6m 59s the patch passed
          +1 compile 7m 33s the patch passed with JDK v1.7.0_91
          +1 javac 7m 33s the patch passed
          -1 checkstyle 0m 18s hadoop-common-project/hadoop-common: patch generated 59 new + 0 unchanged - 0 fixed = 59 total (was 0)
          +1 mvnsite 1m 5s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 13s the patch passed
          +1 javadoc 0m 56s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 6s the patch passed with JDK v1.7.0_91
          +1 unit 7m 59s hadoop-common in the patch passed with JDK v1.8.0_66.
          -1 unit 7m 40s hadoop-common in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          67m 52s



          Reason Tests
          JDK v1.7.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782952/HADOOP-12426-003.patch
          JIRA Issue HADOOP-12426
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 51056509c464 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8bc93db
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8436/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8436/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8436/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8436/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8436/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 6s trunk passed +1 compile 6m 59s trunk passed with JDK v1.8.0_66 +1 compile 7m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 17s trunk passed +1 mvnsite 1m 7s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 59s trunk passed +1 javadoc 0m 59s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 9s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 38s the patch passed +1 compile 6m 59s the patch passed with JDK v1.8.0_66 +1 javac 6m 59s the patch passed +1 compile 7m 33s the patch passed with JDK v1.7.0_91 +1 javac 7m 33s the patch passed -1 checkstyle 0m 18s hadoop-common-project/hadoop-common: patch generated 59 new + 0 unchanged - 0 fixed = 59 total (was 0) +1 mvnsite 1m 5s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 13s the patch passed +1 javadoc 0m 56s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 6s the patch passed with JDK v1.7.0_91 +1 unit 7m 59s hadoop-common in the patch passed with JDK v1.8.0_66. -1 unit 7m 40s hadoop-common in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 67m 52s Reason Tests JDK v1.7.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782952/HADOOP-12426-003.patch JIRA Issue HADOOP-12426 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 51056509c464 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8bc93db Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8436/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8436/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8436/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8436/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8436/console This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          patch -004; fixes all non-indentation related checkstyles

          Show
          stevel@apache.org Steve Loughran added a comment - patch -004; fixes all non-indentation related checkstyles
          Hide
          drankye Kai Zheng added a comment -

          Good work and nice tool! Some comments and hope they're helpful.

          The tool could be improved further as follow-on tasks.
          1. Looks like only Oracle JVM is expected. Not sure how it will behave on other JVMs like IBM JDK.
          2. validateKrb5File could also be supported on Windows, since the krb5 conf file can be retrieved from JAVA_SECURITY_KRB5_CONF. But when it's null, sure it's good to try particularly for non-Windows machines.
          3. A usage() function or the like would be nice to have. I know it's well documented here in the JIRA.
          4. dumpKeytab can dump more than the principal names, information about keys like key type, key version sometimes is also desired.
          5. A try-the-best model might be desired, not aborting immediately when hitting errors, but continuing to find more mismatch issues.
          6. Wonder if it's tool can also be used in client, services and applications, being called at the very beginning, dumping out the troubleshooting messages in the log (security log?). If possible or desired, maybe the dump content can be returned back instead of System.out itself.

          Show
          drankye Kai Zheng added a comment - Good work and nice tool! Some comments and hope they're helpful. The tool could be improved further as follow-on tasks. 1. Looks like only Oracle JVM is expected. Not sure how it will behave on other JVMs like IBM JDK. 2. validateKrb5File could also be supported on Windows, since the krb5 conf file can be retrieved from JAVA_SECURITY_KRB5_CONF. But when it's null, sure it's good to try particularly for non-Windows machines. 3. A usage() function or the like would be nice to have. I know it's well documented here in the JIRA. 4. dumpKeytab can dump more than the principal names, information about keys like key type, key version sometimes is also desired. 5. A try-the-best model might be desired, not aborting immediately when hitting errors, but continuing to find more mismatch issues. 6. Wonder if it's tool can also be used in client, services and applications, being called at the very beginning, dumping out the troubleshooting messages in the log (security log?). If possible or desired, maybe the dump content can be returned back instead of System.out itself.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          thx for the comments.

          1. you should know that I'm stabilising some jenkins-test-run-failures on the slider branch: that test run is failing if there's no default realm, i.e. you are testing on a machine that isn't set up for kerberos.
          2. ...I see what you mean about keytab contents —and that I can get at them. timestamp would be good

          > A try-the-best model might be desired

          I see that...it's already handling the situation where security is off in core-site.xml but has been set on the command line; Checking principals and keytabs is something you can do without worrying about cluster security.

          Maybe the failif() method could be made something that a --nofail option would downgrade to error log; have it return a boolean so that those followon operations which depend on the condition could be skipped.

           if (failif(!keytab.exists(),CAT_CONF, "no keytab %s", keytab)) {
             loginFromkeytab()
          }
          

          of course, I'd have to invert the condition, to something like "require(... )"

          Regarding dumping, there's a --out option which can save it to a file. But as half the log info goes to stderr (all the sun.java stuff), you do need to capture both streams, ideally interleaved. And while I could briefly cache the System.out and System.err streams & replace them with something to catch the output, loggers really hate that.

          As for startup, I think services would need to do the login stuff themselves. You start trying to log in once and not only does UGI lock down, so do bits of the JVM internal state. (that is, UGI.reset() doesn't completely reset things. So I don't think I'd want to have it all there.

          What could be possible?

          • keylength
          • keytab existing
          • dump a keytab
          • look for principal in a keytab
          • All the relevant env vars and properties could be logged
          Show
          stevel@apache.org Steve Loughran added a comment - thx for the comments. you should know that I'm stabilising some jenkins-test-run-failures on the slider branch: that test run is failing if there's no default realm, i.e. you are testing on a machine that isn't set up for kerberos. ...I see what you mean about keytab contents —and that I can get at them. timestamp would be good > A try-the-best model might be desired I see that...it's already handling the situation where security is off in core-site.xml but has been set on the command line; Checking principals and keytabs is something you can do without worrying about cluster security. Maybe the failif() method could be made something that a --nofail option would downgrade to error log; have it return a boolean so that those followon operations which depend on the condition could be skipped. if (failif(!keytab.exists(),CAT_CONF, "no keytab %s" , keytab)) { loginFromkeytab() } of course, I'd have to invert the condition, to something like "require(... )" Regarding dumping, there's a --out option which can save it to a file. But as half the log info goes to stderr (all the sun.java stuff), you do need to capture both streams, ideally interleaved. And while I could briefly cache the System.out and System.err streams & replace them with something to catch the output, loggers really hate that. As for startup, I think services would need to do the login stuff themselves. You start trying to log in once and not only does UGI lock down, so do bits of the JVM internal state. (that is, UGI.reset() doesn't completely reset things. So I don't think I'd want to have it all there. What could be possible? keylength keytab existing dump a keytab look for principal in a keytab All the relevant env vars and properties could be logged
          Hide
          stevel@apache.org Steve Loughran added a comment -

          added

          1. usage
          2. a --nofail option
          3. some printing of the commands
          4. dump of as much of the keytab as is possible
          5. handling of getDefaultRealm failing (including moving it down)
          6. a few more tests

          I can't see what else we can add here, for now

          Show
          stevel@apache.org Steve Loughran added a comment - added usage a --nofail option some printing of the commands dump of as much of the keytab as is possible handling of getDefaultRealm failing (including moving it down) a few more tests I can't see what else we can add here, for now
          Hide
          drankye Kai Zheng added a comment -

          Thanks for the great update! I thought it's pretty quite enough for the initial. The latest patch LGTM.

          Show
          drankye Kai Zheng added a comment - Thanks for the great update! I thought it's pretty quite enough for the initial. The latest patch LGTM.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 5s trunk passed
          +1 compile 9m 17s trunk passed with JDK v1.8.0_66
          +1 compile 9m 13s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 19s trunk passed
          +1 mvnsite 1m 24s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 23s trunk passed
          +1 javadoc 1m 15s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 22s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 51s the patch passed
          +1 compile 10m 28s the patch passed with JDK v1.8.0_66
          +1 javac 10m 28s the patch passed
          +1 compile 9m 35s the patch passed with JDK v1.7.0_91
          +1 javac 9m 35s the patch passed
          -1 checkstyle 0m 20s hadoop-common-project/hadoop-common: patch generated 66 new + 0 unchanged - 0 fixed = 66 total (was 0)
          +1 mvnsite 1m 22s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 2m 51s the patch passed
          +1 javadoc 1m 15s the patch passed with JDK v1.8.0_66
          -1 javadoc 6m 26s hadoop-common-project_hadoop-common-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13)
          +1 javadoc 1m 22s the patch passed with JDK v1.7.0_91
          -1 unit 9m 34s hadoop-common in the patch failed with JDK v1.8.0_66.
          -1 unit 8m 52s hadoop-common in the patch failed with JDK v1.7.0_91.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          93m 55s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag
            hadoop.security.token.delegation.TestZKDelegationTokenSecretManager
            hadoop.ha.TestZKFailoverController
            hadoop.ipc.TestIPC
            hadoop.test.TestTimedOutTestsListener
          JDK v1.7.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783266/HADOOP-12426-006.patch
          JIRA Issue HADOOP-12426
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8c3e21565ef8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1acc509
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/artifact/patchprocess/whitespace-eol.txt
          javadoc hadoop-common-project_hadoop-common-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/artifact/patchprocess/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 5s trunk passed +1 compile 9m 17s trunk passed with JDK v1.8.0_66 +1 compile 9m 13s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 19s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 23s trunk passed +1 javadoc 1m 15s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 22s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 51s the patch passed +1 compile 10m 28s the patch passed with JDK v1.8.0_66 +1 javac 10m 28s the patch passed +1 compile 9m 35s the patch passed with JDK v1.7.0_91 +1 javac 9m 35s the patch passed -1 checkstyle 0m 20s hadoop-common-project/hadoop-common: patch generated 66 new + 0 unchanged - 0 fixed = 66 total (was 0) +1 mvnsite 1m 22s the patch passed +1 mvneclipse 0m 19s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 2m 51s the patch passed +1 javadoc 1m 15s the patch passed with JDK v1.8.0_66 -1 javadoc 6m 26s hadoop-common-project_hadoop-common-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13) +1 javadoc 1m 22s the patch passed with JDK v1.7.0_91 -1 unit 9m 34s hadoop-common in the patch failed with JDK v1.8.0_66. -1 unit 8m 52s hadoop-common in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 93m 55s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag   hadoop.security.token.delegation.TestZKDelegationTokenSecretManager   hadoop.ha.TestZKFailoverController   hadoop.ipc.TestIPC   hadoop.test.TestTimedOutTestsListener JDK v1.7.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783266/HADOOP-12426-006.patch JIRA Issue HADOOP-12426 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8c3e21565ef8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1acc509 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/artifact/patchprocess/whitespace-eol.txt javadoc hadoop-common-project_hadoop-common-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/artifact/patchprocess/diff-javadoc-javadoc-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8440/console This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          got one more feature to add, --resource <resource> (repeatable).

          This will check that a resource is on the classpath (e.g hbase-default.xml, hbase-site.xml) and so help build up the config from that. This will aid in validating that some of the hbase properties are set

          Show
          stevel@apache.org Steve Loughran added a comment - got one more feature to add, --resource <resource> (repeatable). This will check that a resource is on the classpath (e.g hbase-default.xml, hbase-site.xml) and so help build up the config from that. This will aid in validating that some of the hbase properties are set
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch -007

          1. --nologin dump things but skip the login attempts
          2. --resource <xml-resource> add something (e.g hdfs-site.xml) to the config. It's something that has to be on the classpath.
          3. core sysprops extended to include JVM version and vendor
          4. all sysprops are printed, (sorted), afterwards
          5. if /etc/ntp.conf is there, it gets printed (no parsing)

          While getting this working, I encountered the problem where UGI would say "IOE -failed to load", but the underlying messages were being stripped (even though the original stack came through). I've gone through the UGI exception wrapping code and made sure that the previous exception gets included in the error strings. No other changes to UGI exceptions have been made. I know this pulls in a bit of HADOOP-12649, but it was something I needed.

          Show
          stevel@apache.org Steve Loughran added a comment - Patch -007 --nologin dump things but skip the login attempts --resource <xml-resource> add something (e.g hdfs-site.xml) to the config. It's something that has to be on the classpath. core sysprops extended to include JVM version and vendor all sysprops are printed, (sorted), afterwards if /etc/ntp.conf is there, it gets printed (no parsing) While getting this working, I encountered the problem where UGI would say "IOE -failed to load", but the underlying messages were being stripped (even though the original stack came through). I've gone through the UGI exception wrapping code and made sure that the previous exception gets included in the error strings. No other changes to UGI exceptions have been made. I know this pulls in a bit of HADOOP-12649 , but it was something I needed.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          -008 patch, caches IllegalAccessException when trying to reset UGI state. That appears to happen if you've built kdiag into a separate jar and are trying to invoke it. Now it warns it couldn't log in again

          Show
          stevel@apache.org Steve Loughran added a comment - -008 patch, caches IllegalAccessException when trying to reset UGI state. That appears to happen if you've built kdiag into a separate jar and are trying to invoke it. Now it warns it couldn't log in again
          Hide
          stevel@apache.org Steve Loughran added a comment -

          attaching out.txt; full log from the oracle jvm (kdiag now dumps jvm details too). 7u91 login works on oracle

          Show
          stevel@apache.org Steve Loughran added a comment - attaching out.txt; full log from the oracle jvm (kdiag now dumps jvm details too). 7u91 login works on oracle
          Hide
          stevel@apache.org Steve Loughran added a comment -

          (ignore, wrong JIRA. and yes, I am trying to debug kerberos)

          Show
          stevel@apache.org Steve Loughran added a comment - (ignore, wrong JIRA. and yes, I am trying to debug kerberos)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 7m 44s trunk passed
          +1 compile 5m 58s trunk passed with JDK v1.8.0_66
          +1 compile 6m 42s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 45s trunk passed
          +1 javadoc 0m 51s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 3s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 43s the patch passed
          +1 compile 5m 50s the patch passed with JDK v1.8.0_66
          +1 javac 5m 50s the patch passed
          +1 compile 7m 7s the patch passed with JDK v1.7.0_91
          +1 javac 7m 7s the patch passed
          -1 checkstyle 0m 16s hadoop-common-project/hadoop-common: patch generated 78 new + 101 unchanged - 0 fixed = 179 total (was 101)
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          -1 findbugs 2m 4s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 3s the patch passed with JDK v1.7.0_91
          -1 unit 6m 27s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 6m 40s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          60m 37s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            Format string should use %n rather than n in org.apache.hadoop.security.KDiag.arg(String, String, String) At KDiag.java:rather than n in org.apache.hadoop.security.KDiag.arg(String, String, String) At KDiag.java:[line 242]
          JDK v1.8.0_66 Failed junit tests hadoop.test.TestTimedOutTestsListener



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783496/HADOOP-12426-007.patch
          JIRA Issue HADOOP-12426
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux a32b0d01a8f1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1708a4c
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8445/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8445/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/8445/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8445/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8445/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8445/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8445/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 7m 44s trunk passed +1 compile 5m 58s trunk passed with JDK v1.8.0_66 +1 compile 6m 42s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 17s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 0m 51s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 3s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 43s the patch passed +1 compile 5m 50s the patch passed with JDK v1.8.0_66 +1 javac 5m 50s the patch passed +1 compile 7m 7s the patch passed with JDK v1.7.0_91 +1 javac 7m 7s the patch passed -1 checkstyle 0m 16s hadoop-common-project/hadoop-common: patch generated 78 new + 101 unchanged - 0 fixed = 179 total (was 101) +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 0s The patch has no ill-formed XML file. -1 findbugs 2m 4s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 1m 2s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 3s the patch passed with JDK v1.7.0_91 -1 unit 6m 27s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 6m 40s hadoop-common in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 60m 37s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   Format string should use %n rather than n in org.apache.hadoop.security.KDiag.arg(String, String, String) At KDiag.java:rather than n in org.apache.hadoop.security.KDiag.arg(String, String, String) At KDiag.java: [line 242] JDK v1.8.0_66 Failed junit tests hadoop.test.TestTimedOutTestsListener Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783496/HADOOP-12426-007.patch JIRA Issue HADOOP-12426 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux a32b0d01a8f1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1708a4c Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8445/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8445/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/8445/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8445/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8445/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8445/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8445/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 8m 24s trunk passed
          +1 compile 8m 24s trunk passed with JDK v1.8.0_66
          +1 compile 7m 45s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 18s trunk passed
          +1 mvnsite 1m 2s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 58s trunk passed
          +1 javadoc 1m 4s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 21s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 54s the patch passed
          +1 compile 10m 12s the patch passed with JDK v1.8.0_66
          +1 javac 10m 12s the patch passed
          +1 compile 9m 20s the patch passed with JDK v1.7.0_91
          +1 javac 9m 20s the patch passed
          -1 checkstyle 0m 22s hadoop-common-project/hadoop-common: patch generated 78 new + 101 unchanged - 0 fixed = 179 total (was 101)
          +1 mvnsite 1m 17s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          -1 findbugs 2m 41s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 1m 19s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 22s the patch passed with JDK v1.7.0_91
          -1 unit 20m 45s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 9m 48s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 30s Patch does not generate ASF License warnings.
          92m 0s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            Format string should use %n rather than n in org.apache.hadoop.security.KDiag.arg(String, String, String) At KDiag.java:rather than n in org.apache.hadoop.security.KDiag.arg(String, String, String) At KDiag.java:[line 243]
          JDK v1.8.0_66 Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag
            hadoop.fs.TestSymlinkLocalFSFileContext
            hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.test.TestTimedOutTestsListener
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783501/HADOOP-12426-008.patch
          JIRA Issue HADOOP-12426
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 0f6022ade1a4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1708a4c
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8446/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8446/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/8446/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8446/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8446/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8446/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8446/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 8m 24s trunk passed +1 compile 8m 24s trunk passed with JDK v1.8.0_66 +1 compile 7m 45s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 18s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 1m 4s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 21s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 54s the patch passed +1 compile 10m 12s the patch passed with JDK v1.8.0_66 +1 javac 10m 12s the patch passed +1 compile 9m 20s the patch passed with JDK v1.7.0_91 +1 javac 9m 20s the patch passed -1 checkstyle 0m 22s hadoop-common-project/hadoop-common: patch generated 78 new + 101 unchanged - 0 fixed = 179 total (was 101) +1 mvnsite 1m 17s the patch passed +1 mvneclipse 0m 17s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 0s The patch has no ill-formed XML file. -1 findbugs 2m 41s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 1m 19s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 22s the patch passed with JDK v1.7.0_91 -1 unit 20m 45s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 9m 48s hadoop-common in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 30s Patch does not generate ASF License warnings. 92m 0s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   Format string should use %n rather than n in org.apache.hadoop.security.KDiag.arg(String, String, String) At KDiag.java:rather than n in org.apache.hadoop.security.KDiag.arg(String, String, String) At KDiag.java: [line 243] JDK v1.8.0_66 Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag   hadoop.fs.TestSymlinkLocalFSFileContext   hadoop.metrics2.impl.TestGangliaMetrics   hadoop.test.TestTimedOutTestsListener JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12783501/HADOOP-12426-008.patch JIRA Issue HADOOP-12426 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 0f6022ade1a4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1708a4c Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8446/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8446/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/8446/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8446/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8446/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8446/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8446/console This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          finbug check is broken; it's god confused by a \n straight after a %s. Let's see if inserting a space fixes the confusion

            private String arg(String name, String params, String meaning) {
              return String.format("[%s%s%s] : %s\n",
                  name, (!params.isEmpty() ? " " : ""), params, meaning);
            }
          
          Show
          stevel@apache.org Steve Loughran added a comment - finbug check is broken; it's god confused by a \n straight after a %s. Let's see if inserting a space fixes the confusion private String arg( String name, String params, String meaning) { return String .format( "[%s%s%s] : %s\n" , name, (!params.isEmpty() ? " " : ""), params, meaning); }
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch -009

          1. try to make the (spurious) findbugs warning go away
          2. document KDiag in SecureMode.md
          3. improve the usage message
          4. added (commented out) bits to hadoop-env.sh to show how to turn
            on kerberos diagnostics

          With the docs, this is ready to go in. I don't have any immediate plans to extend the tool; if someone can +1 it I'll add it to 2.8

          Show
          stevel@apache.org Steve Loughran added a comment - Patch -009 try to make the (spurious) findbugs warning go away document KDiag in SecureMode.md improve the usage message added (commented out) bits to hadoop-env.sh to show how to turn on kerberos diagnostics With the docs, this is ready to go in. I don't have any immediate plans to extend the tool; if someone can +1 it I'll add it to 2.8
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          0 shelldocs 0m 12s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 7m 2s trunk passed
          +1 compile 7m 20s trunk passed with JDK v1.8.0_66
          +1 compile 7m 35s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 1m 10s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 58s trunk passed
          +1 javadoc 1m 0s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 7s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 45s the patch passed
          +1 compile 7m 4s the patch passed with JDK v1.8.0_66
          +1 javac 7m 4s the patch passed
          +1 compile 7m 41s the patch passed with JDK v1.7.0_91
          +1 javac 7m 41s the patch passed
          -1 checkstyle 0m 24s hadoop-common-project/hadoop-common: patch generated 85 new + 101 unchanged - 0 fixed = 186 total (was 101)
          +1 mvnsite 1m 4s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 shellcheck 0m 9s There were no new shellcheck issues.
          -1 whitespace 0m 0s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 13s the patch passed
          +1 javadoc 0m 56s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 7s the patch passed with JDK v1.7.0_91
          -1 unit 17m 8s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 9m 47s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          78m 9s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784932/HADOOP-12426-009.patch
          JIRA Issue HADOOP-12426
          Optional Tests asflicense mvnsite unit shellcheck shelldocs compile javac javadoc mvninstall findbugs checkstyle xml
          uname Linux d3c786f652fe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 86560a4
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          shellcheck v0.4.3
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8486/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8486/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8486/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8486/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8486/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8486/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. 0 shelldocs 0m 12s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 7m 2s trunk passed +1 compile 7m 20s trunk passed with JDK v1.8.0_66 +1 compile 7m 35s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 23s trunk passed +1 mvnsite 1m 10s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 1m 0s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 7s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 45s the patch passed +1 compile 7m 4s the patch passed with JDK v1.8.0_66 +1 javac 7m 4s the patch passed +1 compile 7m 41s the patch passed with JDK v1.7.0_91 +1 javac 7m 41s the patch passed -1 checkstyle 0m 24s hadoop-common-project/hadoop-common: patch generated 85 new + 101 unchanged - 0 fixed = 186 total (was 101) +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 14s the patch passed +1 shellcheck 0m 9s There were no new shellcheck issues. -1 whitespace 0m 0s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 13s the patch passed +1 javadoc 0m 56s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 7s the patch passed with JDK v1.7.0_91 -1 unit 17m 8s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 9m 47s hadoop-common in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 78m 9s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784932/HADOOP-12426-009.patch JIRA Issue HADOOP-12426 Optional Tests asflicense mvnsite unit shellcheck shelldocs compile javac javadoc mvninstall findbugs checkstyle xml uname Linux d3c786f652fe 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 86560a4 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 shellcheck v0.4.3 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8486/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8486/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8486/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8486/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8486/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8486/console This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          patch -010 address javadoc checkstyle issues.

          I'd really like a review of this and so drop it into 2.8 before the RC is cut. This is inherently backwards compatible going to be invaluable.

          Show
          stevel@apache.org Steve Loughran added a comment - patch -010 address javadoc checkstyle issues. I'd really like a review of this and so drop it into 2.8 before the RC is cut. This is inherently backwards compatible going to be invaluable.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          0 shelldocs 0m 5s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 13s Maven dependency ordering for branch
          +1 mvninstall 6m 45s trunk passed
          +1 compile 5m 49s trunk passed with JDK v1.8.0_72
          +1 compile 6m 41s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 1m 5s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 39s trunk passed
          +1 javadoc 0m 54s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 6s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 42s the patch passed
          +1 compile 5m 47s the patch passed with JDK v1.8.0_72
          +1 javac 5m 47s the patch passed
          +1 compile 6m 38s the patch passed with JDK v1.7.0_95
          +1 javac 6m 38s the patch passed
          -1 checkstyle 0m 23s hadoop-common-project/hadoop-common: patch generated 80 new + 101 unchanged - 0 fixed = 181 total (was 101)
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 shellcheck 0m 9s There were no new shellcheck issues.
          -1 whitespace 0m 0s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 1m 47s the patch passed
          +1 javadoc 0m 52s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 4s the patch passed with JDK v1.7.0_95
          -1 unit 19m 17s hadoop-common in the patch failed with JDK v1.8.0_72.
          +1 unit 8m 17s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          72m 59s



          Reason Tests
          JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786457/HADOOP-12426-010.patch
          JIRA Issue HADOOP-12426
          Optional Tests asflicense mvnsite unit shellcheck shelldocs compile javac javadoc mvninstall findbugs checkstyle xml
          uname Linux 3f6cc4b2d725 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1bcfab8
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          shellcheck v0.4.3
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8546/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8546/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8546/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8546/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8546/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 77MB
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8546/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. 0 shelldocs 0m 5s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 6m 45s trunk passed +1 compile 5m 49s trunk passed with JDK v1.8.0_72 +1 compile 6m 41s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 1m 5s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 39s trunk passed +1 javadoc 0m 54s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 6s trunk passed with JDK v1.7.0_95 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 42s the patch passed +1 compile 5m 47s the patch passed with JDK v1.8.0_72 +1 javac 5m 47s the patch passed +1 compile 6m 38s the patch passed with JDK v1.7.0_95 +1 javac 6m 38s the patch passed -1 checkstyle 0m 23s hadoop-common-project/hadoop-common: patch generated 80 new + 101 unchanged - 0 fixed = 181 total (was 101) +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 15s the patch passed +1 shellcheck 0m 9s There were no new shellcheck issues. -1 whitespace 0m 0s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 52s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 4s the patch passed with JDK v1.7.0_95 -1 unit 19m 17s hadoop-common in the patch failed with JDK v1.8.0_72. +1 unit 8m 17s hadoop-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 72m 59s Reason Tests JDK v1.8.0_72 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12786457/HADOOP-12426-010.patch JIRA Issue HADOOP-12426 Optional Tests asflicense mvnsite unit shellcheck shelldocs compile javac javadoc mvninstall findbugs checkstyle xml uname Linux 3f6cc4b2d725 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1bcfab8 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 shellcheck v0.4.3 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8546/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/8546/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8546/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8546/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8546/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8546/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          all checkstyle tests are related to indentation policy, which, as discussed on common-dev, is not binding. Test failure is clearly unrelated

          Show
          stevel@apache.org Steve Loughran added a comment - all checkstyle tests are related to indentation policy, which, as discussed on common-dev, is not binding. Test failure is clearly unrelated
          Hide
          cnauroth Chris Nauroth added a comment -

          +1. This looks great, Steve. I agree. Let's get what you've already done into 2.8.0 and then iterate further. I'll wait a few hours before committing in case anyone else wants to comment.

          Show
          cnauroth Chris Nauroth added a comment - +1. This looks great, Steve. I agree. Let's get what you've already done into 2.8.0 and then iterate further. I'll wait a few hours before committing in case anyone else wants to comment.
          Hide
          cnauroth Chris Nauroth added a comment -

          I have committed this to trunk, branch-2 and branch-2.8. Steve Loughran, thank you for the contribution. I'm looking forward to using it the next time Kerberos is naughty.

          Show
          cnauroth Chris Nauroth added a comment - I have committed this to trunk, branch-2 and branch-2.8. Steve Loughran , thank you for the contribution. I'm looking forward to using it the next time Kerberos is naughty.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9252 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9252/)
          HADOOP-12426. Add Entry point for Kerberos health check. Contributed by (cnauroth: rev cfa8513890ec1cd7f71258ff3c3083f6d89a4fc4)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/KDiag.java
          • hadoop-common-project/hadoop-common/src/site/markdown/SecureMode.md
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestKDiagNoKDC.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestKDiag.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • hadoop-common-project/hadoop-common/src/test/resources/org/apache/hadoop/security/secure-hdfs-site.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9252 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9252/ ) HADOOP-12426 . Add Entry point for Kerberos health check. Contributed by (cnauroth: rev cfa8513890ec1cd7f71258ff3c3083f6d89a4fc4) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/KDiag.java hadoop-common-project/hadoop-common/src/site/markdown/SecureMode.md hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestKDiagNoKDC.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestKDiag.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java hadoop-common-project/hadoop-common/src/test/resources/org/apache/hadoop/security/secure-hdfs-site.xml

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development