Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: httpfs
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      <!-- markdown -->

      The following environment variables are deprecated. Set the corresponding
      configuration properties instead.

      Environment Variable | Configuration Property | Configuration File
      ----------------------------|------------------------------|--------------------
      HTTPFS_TEMP | hadoop.http.temp.dir | httpfs-site.xml
      HTTPFS_HTTP_PORT | hadoop.httpfs.http.port | httpfs-site.xml
      HTTPFS_MAX_HTTP_HEADER_SIZE | hadoop.http.max.request.header.size and hadoop.http.max.response.header.size | httpfs-site.xml
      HTTPFS_MAX_THREADS | hadoop.http.max.threads | httpfs-site.xml
      HTTPFS_SSL_ENABLED | hadoop.httpfs.ssl.enabled | httpfs-site.xml
      HTTPFS_SSL_KEYSTORE_FILE | ssl.server.keystore.location | ssl-server.xml
      HTTPFS_SSL_KEYSTORE_PASS | ssl.server.keystore.password | ssl-server.xml

      These default HTTP Services have been added.

      Name | Description
      -------------------|------------------------------------
      /conf | Display configuration properties
      /jmx | Java JMX management interface
      /logLevel | Get or set log level per class
      /logs | Display log files
      /stacks | Display JVM stacks
      /static/index.html | The static home page

      Script httpfs.sh has been deprecated, use `hdfs httpfs` instead. The new scripts are based on the Hadoop shell scripting framework. `hadoop daemonlog` is supported. SSL configurations are read from ssl-server.xml.
      Show
      <!-- markdown --> The following environment variables are deprecated. Set the corresponding configuration properties instead. Environment Variable | Configuration Property | Configuration File ----------------------------|------------------------------|-------------------- HTTPFS_TEMP | hadoop.http.temp.dir | httpfs-site.xml HTTPFS_HTTP_PORT | hadoop.httpfs.http.port | httpfs-site.xml HTTPFS_MAX_HTTP_HEADER_SIZE | hadoop.http.max.request.header.size and hadoop.http.max.response.header.size | httpfs-site.xml HTTPFS_MAX_THREADS | hadoop.http.max.threads | httpfs-site.xml HTTPFS_SSL_ENABLED | hadoop.httpfs.ssl.enabled | httpfs-site.xml HTTPFS_SSL_KEYSTORE_FILE | ssl.server.keystore.location | ssl-server.xml HTTPFS_SSL_KEYSTORE_PASS | ssl.server.keystore.password | ssl-server.xml These default HTTP Services have been added. Name | Description -------------------|------------------------------------ /conf | Display configuration properties /jmx | Java JMX management interface /logLevel | Get or set log level per class /logs | Display log files /stacks | Display JVM stacks /static/index.html | The static home page Script httpfs.sh has been deprecated, use `hdfs httpfs` instead. The new scripts are based on the Hadoop shell scripting framework. `hadoop daemonlog` is supported. SSL configurations are read from ssl-server.xml.

      Description

      The Tomcat 6 we are using will reach EOL at the end of 2017. While there are other good options, I would propose switching to Jetty 9 for the following reasons:

      • Easier migration. Both Tomcat and Jetty are based on Servlet Containers, so we don't have to change client code that much. It would require more work to switch to JAX-RS.
      • Well established.
      • Good performance and scalability.

      Other alternatives:

      • Jersey + Grizzly
      • Tomcat 8

      Your opinions will be greatly appreciated.

      1. HDFS-10860.001.patch
        86 kB
        John Zhuge
      2. HDFS-10860.002.patch
        90 kB
        John Zhuge
      3. HDFS-10860.003.patch
        92 kB
        John Zhuge
      4. HDFS-10860.004.patch
        66 kB
        John Zhuge
      5. HDFS-10860.005.patch
        70 kB
        John Zhuge
      6. HDFS-10860.006.patch
        72 kB
        John Zhuge
      7. HDFS-10860.007.patch
        73 kB
        John Zhuge
      8. HDFS-10860.008.patch
        75 kB
        John Zhuge
      9. HDFS-10860.009.patch
        76 kB
        John Zhuge
      10. HDFS-10860.010.patch
        76 kB
        John Zhuge
      11. HDFS-10860.011.patch
        76 kB
        John Zhuge

        Issue Links

          Activity

          Hide
          bobhansen Bob Hansen added a comment -

          Haohui Mai converted the DN side of webhdfs to use Netty for performance and stability. He may have some experience to share.

          Show
          bobhansen Bob Hansen added a comment - Haohui Mai converted the DN side of webhdfs to use Netty for performance and stability. He may have some experience to share.
          Hide
          jzhuge John Zhuge added a comment -

          This is the HttpFS equvalent of HADOOP-13597. Please join discussions over there.

          Show
          jzhuge John Zhuge added a comment - This is the HttpFS equvalent of HADOOP-13597 . Please join discussions over there.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 001

          • Create HttpFSServerWebServer based on HttpServer2
          • Share common code with HADOOP-13597
          • Convert hadoop-hdfs-httpfs from war to jar
          • Update httpfs-default.xml
          • Add configuration keys
          • Load deprecated environment variables and issue warning
          • Add “hdfs httpfs” sub-command
          • Change httpfs.sh to be a wrapper of “hdfs https”
          • Fix MDCFilter NPE when browsing /conf, /logLevel, /jmx, or /stacks
          • Obsolete httpfs.sh version that prints Tomcat version
          • Obsolete HTTP admin port for Tomcat Manager

          TESTING DONE

          • hdfs dfs -ls webhdfs://localhost:14000/
          • hdfs dfs -ls swebhdfs://localhost:14000/“ in SSL mode
          • hdfs https, hdfs —daemon start|status|stop https
          • httpfs.sh run|start|status|stop
          • hadoop daemonlog
          • HttpFS unit tests
          • /jmx, /logLevel, /conf, /stack, /logs, and /static/index.html

          TODO

          • Update docs: ServerSetup.md.vm

          TODO in new JIRAs:

          • Switch unit tests to use HttpFSServerWebServer

          My private branch: https://github.com/jzhuge/hadoop/tree/HDFS-10860.001

          Show
          jzhuge John Zhuge added a comment - Patch 001 Create HttpFSServerWebServer based on HttpServer2 Share common code with HADOOP-13597 Convert hadoop-hdfs-httpfs from war to jar Update httpfs-default.xml Add configuration keys Load deprecated environment variables and issue warning Add “hdfs httpfs” sub-command Change httpfs.sh to be a wrapper of “hdfs https” Fix MDCFilter NPE when browsing /conf, /logLevel, /jmx, or /stacks Obsolete httpfs.sh version that prints Tomcat version Obsolete HTTP admin port for Tomcat Manager TESTING DONE hdfs dfs -ls webhdfs://localhost:14000/ hdfs dfs -ls swebhdfs://localhost:14000/“ in SSL mode hdfs https, hdfs —daemon start|status|stop https httpfs.sh run|start|status|stop hadoop daemonlog HttpFS unit tests /jmx, /logLevel, /conf, /stack, /logs, and /static/index.html TODO Update docs: ServerSetup.md.vm TODO in new JIRAs: Switch unit tests to use HttpFSServerWebServer My private branch: https://github.com/jzhuge/hadoop/tree/HDFS-10860.001
          Hide
          jzhuge John Zhuge added a comment -

          Here is a link to some tests I use: https://github.com/jzhuge/kms-httpfs-test.

          Show
          jzhuge John Zhuge added a comment - Here is a link to some tests I use: https://github.com/jzhuge/kms-httpfs-test .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          0 shelldocs 0m 1s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 1m 54s Maven dependency ordering for branch
          +1 mvninstall 7m 26s trunk passed
          +1 compile 9m 44s trunk passed
          +1 checkstyle 1m 42s trunk passed
          +1 mvnsite 2m 58s trunk passed
          +1 mvneclipse 1m 35s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 2m 56s trunk passed
          +1 javadoc 2m 7s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 1m 52s the patch passed
          +1 compile 10m 57s the patch passed
          -1 javac 10m 57s root generated 4 new + 709 unchanged - 0 fixed = 713 total (was 709)
          +1 checkstyle 1m 45s root: The patch generated 0 new + 90 unchanged - 4 fixed = 90 total (was 94)
          +1 mvnsite 2m 49s the patch passed
          +1 mvneclipse 1m 32s the patch passed
          +1 shellcheck 0m 17s The patch generated 0 new + 568 unchanged - 8 fixed = 568 total (was 576)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 6s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 3m 40s the patch passed
          +1 javadoc 2m 41s the patch passed
          +1 unit 0m 28s hadoop-assemblies in the patch passed.
          -1 unit 4m 40s hadoop-auth in the patch failed.
          -1 unit 9m 16s hadoop-common in the patch failed.
          -1 unit 3m 44s hadoop-hdfs-httpfs in the patch failed.
          +1 asflicense 0m 42s The patch does not generate ASF License warnings.
          77m 25s



          Reason Tests
          Failed junit tests hadoop.security.authentication.client.TestKerberosAuthenticator
            hadoop.security.authentication.client.TestPseudoAuthenticator
            hadoop.security.authentication.server.TestAuthenticationFilter
            hadoop.security.token.delegation.web.TestWebDelegationToken
            hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem
            hadoop.fs.http.server.TestHttpFSWithKerberos
            hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem
            hadoop.fs.http.server.TestHttpFSServerNoACLs
            hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem
            hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem
            hadoop.fs.http.server.TestHttpFSServer
            hadoop.fs.http.server.TestHttpFSServerNoXAttrs



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10860
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842696/HDFS-10860.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle shellcheck shelldocs
          uname Linux 3893e53cb5a4 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4c38f11
          Default Java 1.8.0_111
          shellcheck v0.4.5
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/17822/artifact/patchprocess/diff-compile-javac-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17822/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-auth.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17822/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17822/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17822/testReport/
          modules C: hadoop-assemblies hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17822/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. 0 shelldocs 0m 1s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 1m 54s Maven dependency ordering for branch +1 mvninstall 7m 26s trunk passed +1 compile 9m 44s trunk passed +1 checkstyle 1m 42s trunk passed +1 mvnsite 2m 58s trunk passed +1 mvneclipse 1m 35s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 2m 56s trunk passed +1 javadoc 2m 7s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 1m 52s the patch passed +1 compile 10m 57s the patch passed -1 javac 10m 57s root generated 4 new + 709 unchanged - 0 fixed = 713 total (was 709) +1 checkstyle 1m 45s root: The patch generated 0 new + 90 unchanged - 4 fixed = 90 total (was 94) +1 mvnsite 2m 49s the patch passed +1 mvneclipse 1m 32s the patch passed +1 shellcheck 0m 17s The patch generated 0 new + 568 unchanged - 8 fixed = 568 total (was 576) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 6s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 3m 40s the patch passed +1 javadoc 2m 41s the patch passed +1 unit 0m 28s hadoop-assemblies in the patch passed. -1 unit 4m 40s hadoop-auth in the patch failed. -1 unit 9m 16s hadoop-common in the patch failed. -1 unit 3m 44s hadoop-hdfs-httpfs in the patch failed. +1 asflicense 0m 42s The patch does not generate ASF License warnings. 77m 25s Reason Tests Failed junit tests hadoop.security.authentication.client.TestKerberosAuthenticator   hadoop.security.authentication.client.TestPseudoAuthenticator   hadoop.security.authentication.server.TestAuthenticationFilter   hadoop.security.token.delegation.web.TestWebDelegationToken   hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem   hadoop.fs.http.server.TestHttpFSWithKerberos   hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem   hadoop.fs.http.server.TestHttpFSServerNoACLs   hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem   hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem   hadoop.fs.http.server.TestHttpFSServer   hadoop.fs.http.server.TestHttpFSServerNoXAttrs Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10860 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842696/HDFS-10860.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle shellcheck shelldocs uname Linux 3893e53cb5a4 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4c38f11 Default Java 1.8.0_111 shellcheck v0.4.5 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/17822/artifact/patchprocess/diff-compile-javac-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17822/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-auth.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17822/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17822/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17822/testReport/ modules C: hadoop-assemblies hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17822/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Looking into the test failures caused by my AuthenticationFilter change.

          Show
          jzhuge John Zhuge added a comment - Looking into the test failures caused by my AuthenticationFilter change.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 002

          • Update doc index.md, ServerSetup.md.vm, and HDFSCommands.md
          • Set HttpServer2.Builder#authFilterConfigurationPrefix to integrate with HttpServer2’s secret provider

          TESTING DONE

          • hdfs dfs -ls webhdfs://localhost:14000/
          • hdfs dfs -ls swebhdfs://localhost:14000/“ in SSL mode
          • hdfs https, hdfs —daemon start|status|stop httpfs
          • httpfs.sh run|start|status|stop
          • hadoop daemonlog
          • HttpFS unit tests
          • dist-test nadoop-common and hadoop-hdfs: http://dist-test.cloudera.org/job?job_id=hadoop.jzhuge.1481879379.20252, 10 unrelated test failures.
          • /jmx, /logLevel, /conf, /stack, /logs, and /static/index.html
          Show
          jzhuge John Zhuge added a comment - Patch 002 Update doc index.md, ServerSetup.md.vm, and HDFSCommands.md Set HttpServer2.Builder#authFilterConfigurationPrefix to integrate with HttpServer2’s secret provider TESTING DONE hdfs dfs -ls webhdfs://localhost:14000/ hdfs dfs -ls swebhdfs://localhost:14000/“ in SSL mode hdfs https, hdfs —daemon start|status|stop httpfs httpfs.sh run|start|status|stop hadoop daemonlog HttpFS unit tests dist-test nadoop-common and hadoop-hdfs: http://dist-test.cloudera.org/job?job_id=hadoop.jzhuge.1481879379.20252 , 10 unrelated test failures. /jmx, /logLevel, /conf, /stack, /logs, and /static/index.html
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          0 shelldocs 0m 0s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 2m 2s Maven dependency ordering for branch
          +1 mvninstall 12m 30s trunk passed
          +1 compile 9m 46s trunk passed
          +1 checkstyle 1m 32s trunk passed
          +1 mvnsite 3m 12s trunk passed
          +1 mvneclipse 1m 34s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 4m 14s trunk passed
          +1 javadoc 2m 27s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 2m 18s the patch passed
          +1 compile 9m 19s the patch passed
          -1 javac 9m 19s root generated 4 new + 690 unchanged - 0 fixed = 694 total (was 690)
          +1 checkstyle 1m 31s root: The patch generated 0 new + 97 unchanged - 4 fixed = 97 total (was 101)
          +1 mvnsite 3m 9s the patch passed
          +1 mvneclipse 1m 33s the patch passed
          +1 shellcheck 0m 14s The patch generated 0 new + 566 unchanged - 10 fixed = 566 total (was 576)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 4m 40s the patch passed
          +1 javadoc 2m 26s the patch passed
          +1 unit 0m 15s hadoop-assemblies in the patch passed.
          +1 unit 3m 32s hadoop-auth in the patch passed.
          +1 unit 9m 20s hadoop-common in the patch passed.
          +1 unit 63m 32s hadoop-hdfs in the patch passed.
          +1 unit 3m 2s hadoop-hdfs-httpfs in the patch passed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          144m 47s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10860
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843699/HDFS-10860.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle shellcheck shelldocs
          uname Linux 63a6b69fd85d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / fcbe152
          Default Java 1.8.0_111
          shellcheck v0.4.5
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/17882/artifact/patchprocess/diff-compile-javac-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17882/testReport/
          modules C: hadoop-assemblies hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17882/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. 0 shelldocs 0m 0s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 2m 2s Maven dependency ordering for branch +1 mvninstall 12m 30s trunk passed +1 compile 9m 46s trunk passed +1 checkstyle 1m 32s trunk passed +1 mvnsite 3m 12s trunk passed +1 mvneclipse 1m 34s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 4m 14s trunk passed +1 javadoc 2m 27s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 2m 18s the patch passed +1 compile 9m 19s the patch passed -1 javac 9m 19s root generated 4 new + 690 unchanged - 0 fixed = 694 total (was 690) +1 checkstyle 1m 31s root: The patch generated 0 new + 97 unchanged - 4 fixed = 97 total (was 101) +1 mvnsite 3m 9s the patch passed +1 mvneclipse 1m 33s the patch passed +1 shellcheck 0m 14s The patch generated 0 new + 566 unchanged - 10 fixed = 566 total (was 576) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 4m 40s the patch passed +1 javadoc 2m 26s the patch passed +1 unit 0m 15s hadoop-assemblies in the patch passed. +1 unit 3m 32s hadoop-auth in the patch passed. +1 unit 9m 20s hadoop-common in the patch passed. +1 unit 63m 32s hadoop-hdfs in the patch passed. +1 unit 3m 2s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 144m 47s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10860 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843699/HDFS-10860.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle shellcheck shelldocs uname Linux 63a6b69fd85d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fcbe152 Default Java 1.8.0_111 shellcheck v0.4.5 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/17882/artifact/patchprocess/diff-compile-javac-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17882/testReport/ modules C: hadoop-assemblies hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17882/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 003

          TESTING DONE

          Show
          jzhuge John Zhuge added a comment - Patch 003 Xiao's comments in HADOOP-13597 TESTING DONE Bats regression tests https://github.com/jzhuge/hadoop-regression-tests in unsecure and ssl mode Verify docs
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          0 shelldocs 0m 0s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 20s Maven dependency ordering for branch
          +1 mvninstall 14m 10s trunk passed
          +1 compile 10m 53s trunk passed
          +1 checkstyle 1m 47s trunk passed
          +1 mvnsite 3m 50s trunk passed
          +1 mvneclipse 2m 35s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 5m 29s trunk passed
          +1 javadoc 3m 14s trunk passed
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 2m 56s the patch passed
          +1 compile 11m 43s the patch passed
          -1 javac 11m 43s root generated 4 new + 690 unchanged - 0 fixed = 694 total (was 690)
          +1 checkstyle 1m 54s root: The patch generated 0 new + 95 unchanged - 4 fixed = 95 total (was 99)
          +1 mvnsite 5m 1s the patch passed
          +1 mvneclipse 2m 10s the patch passed
          +1 shellcheck 0m 20s The patch generated 0 new + 565 unchanged - 10 fixed = 565 total (was 575)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 6m 3s the patch passed
          +1 javadoc 3m 19s the patch passed
          +1 unit 0m 24s hadoop-assemblies in the patch passed.
          +1 unit 3m 48s hadoop-auth in the patch passed.
          +1 unit 10m 20s hadoop-common in the patch passed.
          +1 unit 68m 26s hadoop-hdfs in the patch passed.
          +1 unit 3m 14s hadoop-hdfs-httpfs in the patch passed.
          +1 asflicense 0m 39s The patch does not generate ASF License warnings.
          165m 9s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10860
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844464/HDFS-10860.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle shellcheck shelldocs
          uname Linux 17f26936fd8d 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 22befbd
          Default Java 1.8.0_111
          shellcheck v0.4.5
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/17939/artifact/patchprocess/diff-compile-javac-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17939/testReport/
          modules C: hadoop-assemblies hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17939/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. 0 shelldocs 0m 0s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 14m 10s trunk passed +1 compile 10m 53s trunk passed +1 checkstyle 1m 47s trunk passed +1 mvnsite 3m 50s trunk passed +1 mvneclipse 2m 35s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 5m 29s trunk passed +1 javadoc 3m 14s trunk passed 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 2m 56s the patch passed +1 compile 11m 43s the patch passed -1 javac 11m 43s root generated 4 new + 690 unchanged - 0 fixed = 694 total (was 690) +1 checkstyle 1m 54s root: The patch generated 0 new + 95 unchanged - 4 fixed = 95 total (was 99) +1 mvnsite 5m 1s the patch passed +1 mvneclipse 2m 10s the patch passed +1 shellcheck 0m 20s The patch generated 0 new + 565 unchanged - 10 fixed = 565 total (was 575) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 6m 3s the patch passed +1 javadoc 3m 19s the patch passed +1 unit 0m 24s hadoop-assemblies in the patch passed. +1 unit 3m 48s hadoop-auth in the patch passed. +1 unit 10m 20s hadoop-common in the patch passed. +1 unit 68m 26s hadoop-hdfs in the patch passed. +1 unit 3m 14s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 165m 9s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10860 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844464/HDFS-10860.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle shellcheck shelldocs uname Linux 17f26936fd8d 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 22befbd Default Java 1.8.0_111 shellcheck v0.4.5 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/17939/artifact/patchprocess/diff-compile-javac-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17939/testReport/ modules C: hadoop-assemblies hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17939/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks John Zhuge for the patch. Looks good in general - I assume we need a rebase after (IMO almost ready) HADOOP-13597 is committed, will have a closer look at that time.

          Only comment is let's add a very basic test class for the new HttpFSServerWebServer since it's not used anywhere else - start/stop and a simple request/response should do. Can enhance it to cover more later if needed.

          Show
          xiaochen Xiao Chen added a comment - Thanks John Zhuge for the patch. Looks good in general - I assume we need a rebase after (IMO almost ready) HADOOP-13597 is committed, will have a closer look at that time. Only comment is let's add a very basic test class for the new HttpFSServerWebServer since it's not used anywhere else - start/stop and a simple request/response should do. Can enhance it to cover more later if needed.
          Hide
          jzhuge John Zhuge added a comment -

          Thanks Xiao Chen. Yes, we will need a rebase after HADOOP-13597 commit.

          Will add unit tests for HttpFSServerWebServer.

          Show
          jzhuge John Zhuge added a comment - Thanks Xiao Chen . Yes, we will need a rebase after HADOOP-13597 commit. Will add unit tests for HttpFSServerWebServer .
          Hide
          jzhuge John Zhuge added a comment -

          Patch 004

          • Update CommandsManual.md and SecureMode.md

          TESTING DONE

          Show
          jzhuge John Zhuge added a comment - Patch 004 Update CommandsManual.md and SecureMode.md TESTING DONE Bats regression tests https://github.com/jzhuge/hadoop-bats-tests in insecure and ssl mode Verify docs
          Hide
          jzhuge John Zhuge added a comment -

          To add unit tests for HttpFSServerWebServer

          Show
          jzhuge John Zhuge added a comment - To add unit tests for HttpFSServerWebServer
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          0 shelldocs 0m 0s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 58s Maven dependency ordering for branch
          +1 mvninstall 13m 40s trunk passed
          +1 compile 12m 3s trunk passed
          +1 checkstyle 1m 43s trunk passed
          +1 mvnsite 3m 26s trunk passed
          +1 mvneclipse 1m 41s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 4m 47s trunk passed
          +1 javadoc 2m 45s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 2m 28s the patch passed
          +1 compile 11m 26s the patch passed
          +1 javac 11m 26s the patch passed
          +1 checkstyle 1m 51s the patch passed
          +1 mvnsite 3m 59s the patch passed
          +1 mvneclipse 1m 40s the patch passed
          +1 shellcheck 0m 17s The patch generated 0 new + 564 unchanged - 8 fixed = 564 total (was 572)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 6s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 5m 14s the patch passed
          +1 javadoc 2m 47s the patch passed
          +1 unit 0m 22s hadoop-assemblies in the patch passed.
          -1 unit 9m 10s hadoop-common in the patch failed.
          -1 unit 78m 7s hadoop-hdfs in the patch failed.
          +1 unit 3m 57s hadoop-hdfs-httpfs in the patch passed.
          +1 asflicense 0m 49s The patch does not generate ASF License warnings.
          166m 31s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController
          Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10860
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845934/HDFS-10860.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle
          uname Linux 51ec74024e70 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4a659ff
          Default Java 1.8.0_111
          shellcheck v0.4.5
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18045/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18045/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18045/testReport/
          modules C: hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18045/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. 0 shelldocs 0m 0s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 58s Maven dependency ordering for branch +1 mvninstall 13m 40s trunk passed +1 compile 12m 3s trunk passed +1 checkstyle 1m 43s trunk passed +1 mvnsite 3m 26s trunk passed +1 mvneclipse 1m 41s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 4m 47s trunk passed +1 javadoc 2m 45s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 2m 28s the patch passed +1 compile 11m 26s the patch passed +1 javac 11m 26s the patch passed +1 checkstyle 1m 51s the patch passed +1 mvnsite 3m 59s the patch passed +1 mvneclipse 1m 40s the patch passed +1 shellcheck 0m 17s The patch generated 0 new + 564 unchanged - 8 fixed = 564 total (was 572) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 6s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 5m 14s the patch passed +1 javadoc 2m 47s the patch passed +1 unit 0m 22s hadoop-assemblies in the patch passed. -1 unit 9m 10s hadoop-common in the patch failed. -1 unit 78m 7s hadoop-hdfs in the patch failed. +1 unit 3m 57s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 49s The patch does not generate ASF License warnings. 166m 31s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10860 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845934/HDFS-10860.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle uname Linux 51ec74024e70 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4a659ff Default Java 1.8.0_111 shellcheck v0.4.5 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18045/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18045/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18045/testReport/ modules C: hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18045/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 005

          • Add unit tests for HttpFSServerWebServer
          Show
          jzhuge John Zhuge added a comment - Patch 005 Add unit tests for HttpFSServerWebServer
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          0 shelldocs 0m 0s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 19s Maven dependency ordering for branch
          +1 mvninstall 14m 14s trunk passed
          +1 compile 11m 5s trunk passed
          +1 checkstyle 1m 43s trunk passed
          +1 mvnsite 3m 31s trunk passed
          +1 mvneclipse 1m 33s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 3m 59s trunk passed
          +1 javadoc 2m 26s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 2m 2s the patch passed
          +1 compile 9m 16s the patch passed
          +1 javac 9m 16s the patch passed
          +1 checkstyle 1m 34s the patch passed
          +1 mvnsite 3m 28s the patch passed
          +1 mvneclipse 1m 30s the patch passed
          +1 shellcheck 0m 16s The patch generated 0 new + 564 unchanged - 8 fixed = 564 total (was 572)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 4m 18s the patch passed
          +1 javadoc 2m 24s the patch passed
          +1 unit 0m 20s hadoop-assemblies in the patch passed.
          -1 unit 8m 6s hadoop-common in the patch failed.
          -1 unit 72m 2s hadoop-hdfs in the patch failed.
          +1 unit 3m 19s hadoop-hdfs-httpfs in the patch passed.
          +1 asflicense 0m 39s The patch does not generate ASF License warnings.
          150m 17s



          Reason Tests
          Failed junit tests hadoop.net.TestDNS
            hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
          Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
            org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10860
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845965/HDFS-10860.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle
          uname Linux e05df1e25688 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4a659ff
          Default Java 1.8.0_111
          shellcheck v0.4.5
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18048/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18048/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18048/testReport/
          modules C: hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18048/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. 0 shelldocs 0m 0s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 19s Maven dependency ordering for branch +1 mvninstall 14m 14s trunk passed +1 compile 11m 5s trunk passed +1 checkstyle 1m 43s trunk passed +1 mvnsite 3m 31s trunk passed +1 mvneclipse 1m 33s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 3m 59s trunk passed +1 javadoc 2m 26s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 2m 2s the patch passed +1 compile 9m 16s the patch passed +1 javac 9m 16s the patch passed +1 checkstyle 1m 34s the patch passed +1 mvnsite 3m 28s the patch passed +1 mvneclipse 1m 30s the patch passed +1 shellcheck 0m 16s The patch generated 0 new + 564 unchanged - 8 fixed = 564 total (was 572) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 4m 18s the patch passed +1 javadoc 2m 24s the patch passed +1 unit 0m 20s hadoop-assemblies in the patch passed. -1 unit 8m 6s hadoop-common in the patch failed. -1 unit 72m 2s hadoop-hdfs in the patch failed. +1 unit 3m 19s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 150m 17s Reason Tests Failed junit tests hadoop.net.TestDNS   hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10860 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845965/HDFS-10860.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle uname Linux e05df1e25688 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4a659ff Default Java 1.8.0_111 shellcheck v0.4.5 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18048/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18048/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18048/testReport/ modules C: hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18048/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          All failed unit tests pass locally.

          Show
          jzhuge John Zhuge added a comment - All failed unit tests pass locally.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 005 does not suffer from the issue of HADOOP-13961:

          Testing #1 existing source tree

          1. rm -fr ~/.m2/repository/
          2. ( cd hadoop-maven-plugins/ && mvn install )
          3. mvn clean
          4. mvn install -DskipTests -Dmaven.javadoc.skip -Pnative

          Testing #2 new source tree

          1. rm -fr ~/.m2/repository/
          2. mvn install -DskipTests -Dmaven.javadoc.skip -Pnative
          Show
          jzhuge John Zhuge added a comment - Patch 005 does not suffer from the issue of HADOOP-13961 : Testing #1 existing source tree rm -fr ~/.m2/repository/ ( cd hadoop-maven-plugins/ && mvn install ) mvn clean mvn install -DskipTests -Dmaven.javadoc.skip -Pnative Testing #2 new source tree rm -fr ~/.m2/repository/ mvn install -DskipTests -Dmaven.javadoc.skip -Pnative
          Hide
          jzhuge John Zhuge added a comment -

          Wait for HADOOP-13992

          Show
          jzhuge John Zhuge added a comment - Wait for HADOOP-13992
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks John Zhuge for the work.
          Also reviewed HADOOP-13992, close to commit. Please rebase to a version that contains the above fix once it got committed.

          Also 1 nit: TestHttpFSServerWebServer could use a global test timeout.
          Will have a closer look once a rebase is provided.

          Show
          xiaochen Xiao Chen added a comment - Thanks John Zhuge for the work. Also reviewed HADOOP-13992 , close to commit. Please rebase to a version that contains the above fix once it got committed. Also 1 nit: TestHttpFSServerWebServer could use a global test timeout. Will have a closer look once a rebase is provided.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 006

          • Use SSLFactory.readSSLConfiguration introduced by HADOOP-13992
          • Document use of CredentialProvider API in CredentialProviderAPI.md and ServerSetup.md.vm
          • Set TestHttpFSServerWebServer global timeout to 30 seconds

          TESTING DONE

          • TestHttpFSServerWebServer
          • Run HTTPFS_HTTP_PORT=1234 hdfs httpfs to verify the server is running on port 1234
          • Run HTTPFS_SSL_KEYSTORE_PASS=abcd hdfs httpfs to expect wrong password
          • Run HTTPFS_SSL_KEYSTORE_FILE=/tmp/tt hdfs httpfs to expect invalid keystore path
          • HttpFS Bats regression tests https://github.com/jzhuge/hadoop-regression-tests in insecure and ssl mode
          • Verify CredentialProviderAPI.html and ServerSetup.html
          Show
          jzhuge John Zhuge added a comment - Patch 006 Use SSLFactory.readSSLConfiguration introduced by HADOOP-13992 Document use of CredentialProvider API in CredentialProviderAPI.md and ServerSetup.md.vm Set TestHttpFSServerWebServer global timeout to 30 seconds TESTING DONE TestHttpFSServerWebServer Run HTTPFS_HTTP_PORT=1234 hdfs httpfs to verify the server is running on port 1234 Run HTTPFS_SSL_KEYSTORE_PASS=abcd hdfs httpfs to expect wrong password Run HTTPFS_SSL_KEYSTORE_FILE=/tmp/tt hdfs httpfs to expect invalid keystore path HttpFS Bats regression tests https://github.com/jzhuge/hadoop-regression-tests in insecure and ssl mode Verify CredentialProviderAPI.html and ServerSetup.html
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 55s Docker mode activated.
          0 shelldocs 0m 1s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 2m 10s Maven dependency ordering for branch
          +1 mvninstall 13m 17s trunk passed
          +1 compile 13m 4s trunk passed
          +1 checkstyle 1m 37s trunk passed
          +1 mvnsite 3m 11s trunk passed
          +1 mvneclipse 1m 35s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 3m 59s trunk passed
          +1 javadoc 2m 28s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 2m 11s the patch passed
          +1 compile 12m 25s the patch passed
          +1 javac 12m 25s the patch passed
          +1 checkstyle 1m 45s the patch passed
          +1 mvnsite 3m 50s the patch passed
          +1 mvneclipse 1m 35s the patch passed
          +1 shellcheck 0m 17s The patch generated 0 new + 564 unchanged - 8 fixed = 564 total (was 572)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 7s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 4m 38s the patch passed
          +1 javadoc 2m 29s the patch passed
          +1 unit 0m 21s hadoop-assemblies in the patch passed.
          +1 unit 10m 12s hadoop-common in the patch passed.
          -1 unit 90m 58s hadoop-hdfs in the patch failed.
          +1 unit 3m 34s hadoop-hdfs-httpfs in the patch passed.
          +1 asflicense 0m 44s The patch does not generate ASF License warnings.
          179m 13s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10860
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849745/HDFS-10860.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle
          uname Linux 2a5cf34b106a 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ebd4005
          Default Java 1.8.0_121
          shellcheck v0.4.5
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18285/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18285/testReport/
          modules C: hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18285/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 55s Docker mode activated. 0 shelldocs 0m 1s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 2m 10s Maven dependency ordering for branch +1 mvninstall 13m 17s trunk passed +1 compile 13m 4s trunk passed +1 checkstyle 1m 37s trunk passed +1 mvnsite 3m 11s trunk passed +1 mvneclipse 1m 35s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 3m 59s trunk passed +1 javadoc 2m 28s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 2m 11s the patch passed +1 compile 12m 25s the patch passed +1 javac 12m 25s the patch passed +1 checkstyle 1m 45s the patch passed +1 mvnsite 3m 50s the patch passed +1 mvneclipse 1m 35s the patch passed +1 shellcheck 0m 17s The patch generated 0 new + 564 unchanged - 8 fixed = 564 total (was 572) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 7s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 4m 38s the patch passed +1 javadoc 2m 29s the patch passed +1 unit 0m 21s hadoop-assemblies in the patch passed. +1 unit 10m 12s hadoop-common in the patch passed. -1 unit 90m 58s hadoop-hdfs in the patch failed. +1 unit 3m 34s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 44s The patch does not generate ASF License warnings. 179m 13s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10860 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849745/HDFS-10860.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle uname Linux 2a5cf34b106a 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ebd4005 Default Java 1.8.0_121 shellcheck v0.4.5 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18285/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18285/testReport/ modules C: hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18285/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          TestDataNodeOutlierDetectionViaMetrics failure is unrelated and it passed locally. Filed HDFS-11381 for this test failure.

          Show
          jzhuge John Zhuge added a comment - TestDataNodeOutlierDetectionViaMetrics failure is unrelated and it passed locally. Filed HDFS-11381 for this test failure.
          Hide
          xiaochen Xiao Chen added a comment - - edited

          Thanks John Zhuge for revving! Looks pretty close.

          Some nits / questions:

          • HttpFSAuthenticationFilter.java: CONF_PREFIX can be package private
          • MDCFilter:java: why do we need to check getPathInfo() being null here? Do we need to check other params?
          • ServerSetup.md.vm:
            • NOTE: The script `httpfs.sh` is deprecated. It is now just a wrapper of `hdfs httpfs`. I think it's more accurate to say it's a wrapper of hdfs (which is the script being called)?
            • keytool -genkey -alias tomcat -keyalg RSA let's say bye-bye to tomcat...
            • It seems the paragraph under the keytool line doesn't need the change. Since the example is given 'As the `httpfs` Unix user'.
            • Let's also have a note in the doc that tomcat is deprecated, together with its old .sh configurations. We can point to the Deprecated Environment Variables section you added later, to give more context.
            • In Deprecated Environment Variables, please move HTTPFS_TEMP to the first line to be consistent with the config code, and to better group the 'Configuration File'
          • TestHttpFSServerWebServer.java: I was thinking to add a basic test, to do a simple request after startup, to make sure the server is reachable.
          • index.html: This line doesn't look right /webhdfs/v1/?op=LISTSTATUS to list all keys
          • Sorry should have asked this in the KMS jira. The new index.html page for loglevel/jmx/conf/logs looks nice and handy. But how does the security work here? Have you tested this in a kerberized environment? I tried locally in a pseudo-authenticated setup, it seems I can read/set everything even without the user.name= param.
          Show
          xiaochen Xiao Chen added a comment - - edited Thanks John Zhuge for revving! Looks pretty close. Some nits / questions: HttpFSAuthenticationFilter.java: CONF_PREFIX can be package private MDCFilter:java: why do we need to check getPathInfo() being null here? Do we need to check other params? ServerSetup.md.vm: NOTE: The script `httpfs.sh` is deprecated. It is now just a wrapper of `hdfs httpfs` . I think it's more accurate to say it's a wrapper of hdfs (which is the script being called)? keytool -genkey -alias tomcat -keyalg RSA let's say bye-bye to tomcat... It seems the paragraph under the keytool line doesn't need the change. Since the example is given 'As the `httpfs` Unix user'. Let's also have a note in the doc that tomcat is deprecated, together with its old .sh configurations. We can point to the Deprecated Environment Variables section you added later, to give more context. In Deprecated Environment Variables , please move HTTPFS_TEMP to the first line to be consistent with the config code, and to better group the 'Configuration File' TestHttpFSServerWebServer.java: I was thinking to add a basic test, to do a simple request after startup, to make sure the server is reachable. index.html: This line doesn't look right /webhdfs/v1/?op=LISTSTATUS to list all keys Sorry should have asked this in the KMS jira. The new index.html page for loglevel/jmx/conf/logs looks nice and handy. But how does the security work here? Have you tested this in a kerberized environment? I tried locally in a pseudo-authenticated setup, it seems I can read/set everything even without the user.name= param.
          Hide
          jzhuge John Zhuge added a comment -

          Thanks Xiao Chen for the review.

          HttpFSAuthenticationFilter.java: CONF_PREFIX can be package private

          Fixed

          MDCFilter:java: why do we need to check getPathInfo() being null here? Do we need to check other params?

          GOT NPE without the check for daemonlog request

          ServerSetup.md.vm:

          NOTE: The script `httpfs.sh` is deprecated. It is now just a wrapper of `hdfs httpfs`. I think it's more accurate to say it's a wrapper of hdfs (which is the script being called)?

          `httpfs.sh` is a wrapper of `hdfs httpfs`, not `hdfs` which has many other sub-commands.

          keytool -genkey -alias tomcat -keyalg RSA let's say bye-bye to tomcat...

          Fixed

          It seems the paragraph under the keytool line doesn't need the change. Since the example is given 'As the `httpfs` Unix user'.

          Fixed

          Let's also have a note in the doc that tomcat is deprecated, together with its old .sh configurations. We can point to the Deprecated Environment Variables section you added later, to give more context.

          Fixed in section 'Start/Stop HttpFS'

          In Deprecated Environment Variables, please move HTTPFS_TEMP to the first line to be consistent with the config code, and to better group the 'Configuration File'

          Variables are sorted by name but ok with your suggestion.

          TestHttpFSServerWebServer.java: I was thinking to add a basic test, to do a simple request after startup, to make sure the server is reachable.

          Added to TestStartStop

          index.html: This line doesn't look right /webhdfs/v1/?op=LISTSTATUS to list all keys

          Fixed

          Sorry should have asked this in the KMS jira. The new index.html page for loglevel/jmx/conf/logs looks nice and handy. But how does the security work here? Have you tested this in a kerberized environment? I tried locally in a pseudo-authenticated setup, it seems I can read/set everything even without the user.name= param.

          Will investigate

          Show
          jzhuge John Zhuge added a comment - Thanks Xiao Chen for the review. HttpFSAuthenticationFilter.java: CONF_PREFIX can be package private Fixed MDCFilter:java: why do we need to check getPathInfo() being null here? Do we need to check other params? GOT NPE without the check for daemonlog request ServerSetup.md.vm: NOTE: The script `httpfs.sh` is deprecated. It is now just a wrapper of `hdfs httpfs`. I think it's more accurate to say it's a wrapper of hdfs (which is the script being called)? `httpfs.sh` is a wrapper of `hdfs httpfs`, not `hdfs` which has many other sub-commands. keytool -genkey -alias tomcat -keyalg RSA let's say bye-bye to tomcat... Fixed It seems the paragraph under the keytool line doesn't need the change. Since the example is given 'As the `httpfs` Unix user'. Fixed Let's also have a note in the doc that tomcat is deprecated, together with its old .sh configurations. We can point to the Deprecated Environment Variables section you added later, to give more context. Fixed in section 'Start/Stop HttpFS' In Deprecated Environment Variables, please move HTTPFS_TEMP to the first line to be consistent with the config code, and to better group the 'Configuration File' Variables are sorted by name but ok with your suggestion. TestHttpFSServerWebServer.java: I was thinking to add a basic test, to do a simple request after startup, to make sure the server is reachable. Added to TestStartStop index.html: This line doesn't look right /webhdfs/v1/?op=LISTSTATUS to list all keys Fixed Sorry should have asked this in the KMS jira. The new index.html page for loglevel/jmx/conf/logs looks nice and handy. But how does the security work here? Have you tested this in a kerberized environment? I tried locally in a pseudo-authenticated setup, it seems I can read/set everything even without the user.name= param. Will investigate
          Hide
          jzhuge John Zhuge added a comment -

          MDCFilter:java: why do we need to check getPathInfo() being null here? Do we need to check other params?

          No, I was wrong about NPE without the check for daemonlog request. Will investigate further,

          Show
          jzhuge John Zhuge added a comment - MDCFilter:java: why do we need to check getPathInfo() being null here? Do we need to check other params? No, I was wrong about NPE without the check for daemonlog request. Will investigate further,
          Hide
          jzhuge John Zhuge added a comment -

          Confirmed NPE without the check for daemonlog request, and for /jmx, /conf, and /logLevel.

          Show
          jzhuge John Zhuge added a comment - Confirmed NPE without the check for daemonlog request, and for /jmx, /conf, and /logLevel.
          Hide
          jzhuge John Zhuge added a comment -

          Sorry should have asked this in the KMS jira. The new index.html page for loglevel/jmx/conf/logs looks nice and handy. But how does the security work here? Have you tested this in a kerberized environment? I tried locally in a pseudo-authenticated setup, it seems I can read/set everything even without the user.name= param.

          /conf, /jmx, /logLevel, and /stacks do report "Authentication required" when user.name is not specified.

          /logs do not require authentication. This does not seem right, will file another JIRA to follow up.

          Show
          jzhuge John Zhuge added a comment - Sorry should have asked this in the KMS jira. The new index.html page for loglevel/jmx/conf/logs looks nice and handy. But how does the security work here? Have you tested this in a kerberized environment? I tried locally in a pseudo-authenticated setup, it seems I can read/set everything even without the user.name= param. /conf, /jmx, /logLevel, and /stacks do report "Authentication required" when user.name is not specified. /logs do not require authentication. This does not seem right, will file another JIRA to follow up.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 007

          • Xiao’s comments
          • Test simple request in TestHttpFSServerWebServer#testStartStop

          TESTING DONE

          Show
          jzhuge John Zhuge added a comment - Patch 007 Xiao’s comments Test simple request in TestHttpFSServerWebServer#testStartStop TESTING DONE TestHttpFSServerWebServer HttpFS Bats regression tests https://github.com/jzhuge/hadoop-regression-tests in insecure and ssl mode Verify ServerSetup.html and static index.html
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks John Zhuge.

          As discussed, there're really 2 issues with HttpServer2 security:

          1. authentication should be needed to access everything
          2. based on 1, what level of authorization should be there for each of these servlet.

          Looking at HttpServer2 code, it seems there's a hasAdministratorAccess method. So possibly we just need some configurations set. Will play with it a bit.

          Show
          xiaochen Xiao Chen added a comment - Thanks John Zhuge . As discussed, there're really 2 issues with HttpServer2 security: authentication should be needed to access everything based on 1, what level of authorization should be there for each of these servlet. Looking at HttpServer2 code, it seems there's a hasAdministratorAccess method. So possibly we just need some configurations set. Will play with it a bit.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          0 shelldocs 0m 2s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 59s Maven dependency ordering for branch
          +1 mvninstall 13m 2s trunk passed
          +1 compile 13m 26s trunk passed
          +1 checkstyle 1m 42s trunk passed
          +1 mvnsite 3m 28s trunk passed
          +1 mvneclipse 1m 32s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 3m 54s trunk passed
          +1 javadoc 2m 24s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 5s the patch passed
          +1 compile 11m 31s the patch passed
          +1 javac 11m 31s the patch passed
          +1 checkstyle 1m 40s the patch passed
          +1 mvnsite 3m 27s the patch passed
          +1 mvneclipse 1m 32s the patch passed
          +1 shellcheck 0m 18s The patch generated 0 new + 564 unchanged - 8 fixed = 564 total (was 572)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 5s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 9m 59s the patch passed
          +1 javadoc 5m 5s the patch passed
          +1 unit 0m 23s hadoop-assemblies in the patch passed.
          +1 unit 9m 3s hadoop-common in the patch passed.
          +1 unit 63m 20s hadoop-hdfs in the patch passed.
          +1 unit 3m 18s hadoop-hdfs-httpfs in the patch passed.
          +1 asflicense 0m 40s The patch does not generate ASF License warnings.
          158m 58s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10860
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850322/HDFS-10860.007.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle
          uname Linux d1c7e6bf1c6b 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 258991d
          Default Java 1.8.0_121
          shellcheck v0.4.5
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18300/testReport/
          modules C: hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18300/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. 0 shelldocs 0m 2s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 59s Maven dependency ordering for branch +1 mvninstall 13m 2s trunk passed +1 compile 13m 26s trunk passed +1 checkstyle 1m 42s trunk passed +1 mvnsite 3m 28s trunk passed +1 mvneclipse 1m 32s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 3m 54s trunk passed +1 javadoc 2m 24s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 5s the patch passed +1 compile 11m 31s the patch passed +1 javac 11m 31s the patch passed +1 checkstyle 1m 40s the patch passed +1 mvnsite 3m 27s the patch passed +1 mvneclipse 1m 32s the patch passed +1 shellcheck 0m 18s The patch generated 0 new + 564 unchanged - 8 fixed = 564 total (was 572) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 5s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 9m 59s the patch passed +1 javadoc 5m 5s the patch passed +1 unit 0m 23s hadoop-assemblies in the patch passed. +1 unit 9m 3s hadoop-common in the patch passed. +1 unit 63m 20s hadoop-hdfs in the patch passed. +1 unit 3m 18s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 40s The patch does not generate ASF License warnings. 158m 58s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10860 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850322/HDFS-10860.007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle uname Linux d1c7e6bf1c6b 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 258991d Default Java 1.8.0_121 shellcheck v0.4.5 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18300/testReport/ modules C: hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18300/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          HttpServer2#isInstrumentationAccessAllowed requires the following conditions to return false:

          • hadoop.security.instrumentation.requires.admin set to true
          • hadoop.security.authorization set to true
          • remote user is not set or HttpServer2#setACL is called with an ACL that disallows the remote user

          Looks like we need another configuration property to set administrators for HttpFS server instrumentation access, e.g., hadoop.httpfs.http.administrators, similar to dfs.cluster.administrators.

          Will need to fix KMS to call HttpServer2#setACL. Also gotta make sure all users of HttpServer2 call setACL.

          Show
          jzhuge John Zhuge added a comment - HttpServer2#isInstrumentationAccessAllowed requires the following conditions to return false: hadoop.security.instrumentation.requires.admin set to true hadoop.security.authorization set to true remote user is not set or HttpServer2#setACL is called with an ACL that disallows the remote user Looks like we need another configuration property to set administrators for HttpFS server instrumentation access, e.g., hadoop.httpfs.http.administrators , similar to dfs.cluster.administrators . Will need to fix KMS to call HttpServer2#setACL . Also gotta make sure all users of HttpServer2 call setACL .
          Hide
          jzhuge John Zhuge added a comment -

          Patch 008

          • Add config property hadoop.httpfs.http.administrators to control access to instrumentation servlets
          • Enhance TestHttpFSServerWebServer to start server on any port of localhost

          TESTING DONE

          Show
          jzhuge John Zhuge added a comment - Patch 008 Add config property hadoop.httpfs.http.administrators to control access to instrumentation servlets Enhance TestHttpFSServerWebServer to start server on any port of localhost TESTING DONE TestHttpFSServerWebServer HttpFS Bats regression tests https://github.com/jzhuge/hadoop-regression-tests in insecure and ssl mode Verify ServerSetup.html and static index.html
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          0 shelldocs 0m 1s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 55s Maven dependency ordering for branch
          +1 mvninstall 15m 54s trunk passed
          +1 compile 15m 10s trunk passed
          +1 checkstyle 1m 54s trunk passed
          +1 mvnsite 4m 0s trunk passed
          +1 mvneclipse 1m 41s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 4m 56s trunk passed
          +1 javadoc 2m 54s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 2m 41s the patch passed
          +1 compile 12m 56s the patch passed
          +1 javac 12m 56s the patch passed
          +1 checkstyle 1m 47s the patch passed
          +1 mvnsite 8m 57s the patch passed
          +1 mvneclipse 6m 53s the patch passed
          +1 shellcheck 0m 26s The patch generated 0 new + 564 unchanged - 8 fixed = 564 total (was 572)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 15s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 8m 1s the patch passed
          +1 javadoc 2m 44s the patch passed
          +1 unit 0m 21s hadoop-assemblies in the patch passed.
          -1 unit 9m 21s hadoop-common in the patch failed.
          -1 unit 83m 35s hadoop-hdfs in the patch failed.
          +1 unit 4m 7s hadoop-hdfs-httpfs in the patch passed.
          +1 asflicense 0m 57s The patch does not generate ASF License warnings.
          193m 59s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController
          Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10860
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850381/HDFS-10860.008.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle
          uname Linux 56eb969f3740 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bec9b7a
          Default Java 1.8.0_121
          shellcheck v0.4.5
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18304/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18304/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18304/testReport/
          modules C: hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18304/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. 0 shelldocs 0m 1s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 55s Maven dependency ordering for branch +1 mvninstall 15m 54s trunk passed +1 compile 15m 10s trunk passed +1 checkstyle 1m 54s trunk passed +1 mvnsite 4m 0s trunk passed +1 mvneclipse 1m 41s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 4m 56s trunk passed +1 javadoc 2m 54s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 2m 41s the patch passed +1 compile 12m 56s the patch passed +1 javac 12m 56s the patch passed +1 checkstyle 1m 47s the patch passed +1 mvnsite 8m 57s the patch passed +1 mvneclipse 6m 53s the patch passed +1 shellcheck 0m 26s The patch generated 0 new + 564 unchanged - 8 fixed = 564 total (was 572) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 15s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 8m 1s the patch passed +1 javadoc 2m 44s the patch passed +1 unit 0m 21s hadoop-assemblies in the patch passed. -1 unit 9m 21s hadoop-common in the patch failed. -1 unit 83m 35s hadoop-hdfs in the patch failed. +1 unit 4m 7s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 57s The patch does not generate ASF License warnings. 193m 59s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10860 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850381/HDFS-10860.008.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle uname Linux 56eb969f3740 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bec9b7a Default Java 1.8.0_121 shellcheck v0.4.5 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18304/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18304/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18304/testReport/ modules C: hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18304/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks John Zhuge, patch 8 looks good to me. Will let this soak for a couple of days in case other audience have comments, and recheck on Friday.

          Can you open the 2 jiras for:

          • servlet admin control for kms, as a follow-up of HADOOP-13597.
          • research why HttpServer2's logs are not following the same security model. I looked back on that class's history, but no clue so far.
          Show
          xiaochen Xiao Chen added a comment - Thanks John Zhuge , patch 8 looks good to me. Will let this soak for a couple of days in case other audience have comments, and recheck on Friday. Can you open the 2 jiras for: servlet admin control for kms, as a follow-up of HADOOP-13597 . research why HttpServer2's logs are not following the same security model. I looked back on that class's history, but no clue so far.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 009

          • Update ServerSetup.md.vm

          TESTING DONE

          • Verify ServerSetup.html
          Show
          jzhuge John Zhuge added a comment - Patch 009 Update ServerSetup.md.vm TESTING DONE Verify ServerSetup.html
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          0 shelldocs 0m 1s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 2m 16s Maven dependency ordering for branch
          +1 mvninstall 14m 31s trunk passed
          +1 compile 14m 3s trunk passed
          +1 checkstyle 1m 42s trunk passed
          +1 mvnsite 3m 22s trunk passed
          +1 mvneclipse 1m 33s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 4m 26s trunk passed
          +1 javadoc 2m 34s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 22s the patch passed
          +1 compile 13m 4s the patch passed
          +1 javac 13m 4s the patch passed
          +1 checkstyle 1m 44s the patch passed
          +1 mvnsite 3m 39s the patch passed
          +1 mvneclipse 1m 28s the patch passed
          +1 shellcheck 0m 19s The patch generated 0 new + 564 unchanged - 8 fixed = 564 total (was 572)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 4m 59s the patch passed
          +1 javadoc 2m 40s the patch passed
          +1 unit 0m 22s hadoop-assemblies in the patch passed.
          -1 unit 9m 52s hadoop-common in the patch failed.
          -1 unit 75m 46s hadoop-hdfs in the patch failed.
          +1 unit 3m 51s hadoop-hdfs-httpfs in the patch passed.
          +1 asflicense 0m 40s The patch does not generate ASF License warnings.
          167m 26s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.security.TestKDiag
            hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10860
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850510/HDFS-10860.009.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle
          uname Linux 83c416a01771 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b6f290d
          Default Java 1.8.0_121
          shellcheck v0.4.5
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18311/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18311/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18311/testReport/
          modules C: hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18311/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. 0 shelldocs 0m 1s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 2m 16s Maven dependency ordering for branch +1 mvninstall 14m 31s trunk passed +1 compile 14m 3s trunk passed +1 checkstyle 1m 42s trunk passed +1 mvnsite 3m 22s trunk passed +1 mvneclipse 1m 33s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 4m 26s trunk passed +1 javadoc 2m 34s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 22s the patch passed +1 compile 13m 4s the patch passed +1 javac 13m 4s the patch passed +1 checkstyle 1m 44s the patch passed +1 mvnsite 3m 39s the patch passed +1 mvneclipse 1m 28s the patch passed +1 shellcheck 0m 19s The patch generated 0 new + 564 unchanged - 8 fixed = 564 total (was 572) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 4m 59s the patch passed +1 javadoc 2m 40s the patch passed +1 unit 0m 22s hadoop-assemblies in the patch passed. -1 unit 9m 52s hadoop-common in the patch failed. -1 unit 75m 46s hadoop-hdfs in the patch failed. +1 unit 3m 51s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 40s The patch does not generate ASF License warnings. 167m 26s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.security.TestKDiag   hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10860 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850510/HDFS-10860.009.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle uname Linux 83c416a01771 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b6f290d Default Java 1.8.0_121 shellcheck v0.4.5 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18311/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18311/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18311/testReport/ modules C: hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18311/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Upgrading this to a blocker for alpha3, looks like we're close in any case but this is something I definitely want in.

          Show
          andrew.wang Andrew Wang added a comment - Upgrading this to a blocker for alpha3, looks like we're close in any case but this is something I definitely want in.
          Hide
          jzhuge John Zhuge added a comment -

          Filed 2 JIRAs:

          • HADOOP-14047 Require admin to access KMS instrumentation servlets
          • HADOOP-14060 KMS /logs servlet should have access control
          Show
          jzhuge John Zhuge added a comment - Filed 2 JIRAs: HADOOP-14047 Require admin to access KMS instrumentation servlets HADOOP-14060 KMS /logs servlet should have access control
          Hide
          jzhuge John Zhuge added a comment -

          Patch 010

          • Update ServerSetup.md.vm

          TESTING DONE

          Show
          jzhuge John Zhuge added a comment - Patch 010 Update ServerSetup.md.vm TESTING DONE - Verify /jmx, /logLevel, /conf, and /stack with hadoop.httpfs.http.administrators set to “$USER” No access for kerberos login "hdfs" Access for kerberos login “$USER" HttpFS Bats regression tests https://github.com/jzhuge/hadoop-bats-tests and https://github.com/jzhuge/hadoop-setup-scripts in insecure, ssl, and ssl+kerberos mode
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          0 shelldocs 0m 0s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 57s Maven dependency ordering for branch
          +1 mvninstall 14m 43s trunk passed
          +1 compile 14m 35s trunk passed
          +1 checkstyle 1m 53s trunk passed
          +1 mvnsite 3m 55s trunk passed
          +1 mvneclipse 1m 36s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 4m 58s trunk passed
          +1 javadoc 2m 41s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 2m 24s the patch passed
          +1 compile 11m 48s the patch passed
          +1 javac 11m 48s the patch passed
          +1 checkstyle 1m 48s the patch passed
          +1 mvnsite 4m 4s the patch passed
          +1 mvneclipse 1m 43s the patch passed
          +1 shellcheck 0m 18s The patch generated 0 new + 564 unchanged - 8 fixed = 564 total (was 572)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 6s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 5m 17s the patch passed
          +1 javadoc 2m 52s the patch passed
          +1 unit 0m 21s hadoop-assemblies in the patch passed.
          -1 unit 9m 4s hadoop-common in the patch failed.
          -1 unit 79m 56s hadoop-hdfs in the patch failed.
          +1 unit 4m 17s hadoop-hdfs-httpfs in the patch passed.
          +1 asflicense 0m 46s The patch does not generate ASF License warnings.
          173m 26s



          Reason Tests
          Failed junit tests hadoop.security.TestGroupsCaching
            hadoop.hdfs.server.datanode.TestFsDatasetCache
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10860
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12851128/HDFS-10860.010.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle
          uname Linux d03fd68431d8 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9cbbd1e
          Default Java 1.8.0_121
          shellcheck v0.4.5
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18330/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18330/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18330/testReport/
          modules C: hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18330/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. 0 shelldocs 0m 0s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 57s Maven dependency ordering for branch +1 mvninstall 14m 43s trunk passed +1 compile 14m 35s trunk passed +1 checkstyle 1m 53s trunk passed +1 mvnsite 3m 55s trunk passed +1 mvneclipse 1m 36s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 4m 58s trunk passed +1 javadoc 2m 41s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 2m 24s the patch passed +1 compile 11m 48s the patch passed +1 javac 11m 48s the patch passed +1 checkstyle 1m 48s the patch passed +1 mvnsite 4m 4s the patch passed +1 mvneclipse 1m 43s the patch passed +1 shellcheck 0m 18s The patch generated 0 new + 564 unchanged - 8 fixed = 564 total (was 572) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 6s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 5m 17s the patch passed +1 javadoc 2m 52s the patch passed +1 unit 0m 21s hadoop-assemblies in the patch passed. -1 unit 9m 4s hadoop-common in the patch failed. -1 unit 79m 56s hadoop-hdfs in the patch failed. +1 unit 4m 17s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 46s The patch does not generate ASF License warnings. 173m 26s Reason Tests Failed junit tests hadoop.security.TestGroupsCaching   hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10860 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12851128/HDFS-10860.010.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle uname Linux d03fd68431d8 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9cbbd1e Default Java 1.8.0_121 shellcheck v0.4.5 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18330/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18330/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18330/testReport/ modules C: hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18330/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 011

          • Fix a typo in patch 010
          Show
          jzhuge John Zhuge added a comment - Patch 011 Fix a typo in patch 010
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for revving John! +1 pending jenkins.

          Show
          xiaochen Xiao Chen added a comment - Thanks for revving John! +1 pending jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          0 shelldocs 0m 1s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 2m 23s Maven dependency ordering for branch
          +1 mvninstall 14m 15s trunk passed
          +1 compile 13m 49s trunk passed
          +1 checkstyle 1m 39s trunk passed
          +1 mvnsite 3m 12s trunk passed
          +1 mvneclipse 1m 38s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 4m 4s trunk passed
          +1 javadoc 2m 34s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 2m 11s the patch passed
          +1 compile 12m 0s the patch passed
          +1 javac 12m 0s the patch passed
          +1 checkstyle 1m 41s the patch passed
          +1 mvnsite 4m 15s the patch passed
          +1 mvneclipse 1m 54s the patch passed
          +1 shellcheck 0m 18s The patch generated 0 new + 562 unchanged - 8 fixed = 562 total (was 570)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 6s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies
          +1 findbugs 5m 15s the patch passed
          +1 javadoc 2m 53s the patch passed
          +1 unit 0m 24s hadoop-assemblies in the patch passed.
          +1 unit 10m 31s hadoop-common in the patch passed.
          -1 unit 105m 17s hadoop-hdfs in the patch failed.
          +1 unit 3m 28s hadoop-hdfs-httpfs in the patch passed.
          +1 asflicense 0m 41s The patch does not generate ASF License warnings.
          196m 52s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10860
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12851241/HDFS-10860.011.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle
          uname Linux 3be351e58bdd 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d88497d
          Default Java 1.8.0_121
          shellcheck v0.4.5
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18334/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18334/testReport/
          modules C: hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18334/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. 0 shelldocs 0m 1s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 2m 23s Maven dependency ordering for branch +1 mvninstall 14m 15s trunk passed +1 compile 13m 49s trunk passed +1 checkstyle 1m 39s trunk passed +1 mvnsite 3m 12s trunk passed +1 mvneclipse 1m 38s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 4m 4s trunk passed +1 javadoc 2m 34s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 2m 11s the patch passed +1 compile 12m 0s the patch passed +1 javac 12m 0s the patch passed +1 checkstyle 1m 41s the patch passed +1 mvnsite 4m 15s the patch passed +1 mvneclipse 1m 54s the patch passed +1 shellcheck 0m 18s The patch generated 0 new + 562 unchanged - 8 fixed = 562 total (was 570) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 6s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-assemblies +1 findbugs 5m 15s the patch passed +1 javadoc 2m 53s the patch passed +1 unit 0m 24s hadoop-assemblies in the patch passed. +1 unit 10m 31s hadoop-common in the patch passed. -1 unit 105m 17s hadoop-hdfs in the patch failed. +1 unit 3m 28s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 41s The patch does not generate ASF License warnings. 196m 52s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10860 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12851241/HDFS-10860.011.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml shellcheck shelldocs findbugs checkstyle uname Linux 3be351e58bdd 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d88497d Default Java 1.8.0_121 shellcheck v0.4.5 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18334/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18334/testReport/ modules C: hadoop-assemblies hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18334/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          TestDataNodeVolumeFailureReporting, TestNameNodeMetadataConsistency, TestThrottledAsyncChecker, TestRollingUpgrade, TestDataNodeVolumeFailure failures are unrelated and they pass locally for me.

          Show
          jzhuge John Zhuge added a comment - TestDataNodeVolumeFailureReporting, TestNameNodeMetadataConsistency, TestThrottledAsyncChecker, TestRollingUpgrade, TestDataNodeVolumeFailure failures are unrelated and they pass locally for me.
          Hide
          jzhuge John Zhuge added a comment - - edited

          Filed HDFS-11396 TestNameNodeMetadataConsistency#testGenerationStampInFuture timed out.

          Show
          jzhuge John Zhuge added a comment - - edited Filed HDFS-11396 TestNameNodeMetadataConsistency#testGenerationStampInFuture timed out.
          Hide
          jzhuge John Zhuge added a comment - - edited

          Filed HDFS-11397 TestThrottledAsyncChecker#testCancellation timed out.

          Show
          jzhuge John Zhuge added a comment - - edited Filed HDFS-11397 TestThrottledAsyncChecker#testCancellation timed out.
          Hide
          xiaochen Xiao Chen added a comment -

          +1 on patch 11, committing.

          Show
          xiaochen Xiao Chen added a comment - +1 on patch 11, committing.
          Hide
          xiaochen Xiao Chen added a comment -

          Committed to trunk! Thanks John Zhuge for the great work here, and Bob Hansen for the early pointer.

          Show
          xiaochen Xiao Chen added a comment - Committed to trunk! Thanks John Zhuge for the great work here, and Bob Hansen for the early pointer.
          Hide
          xiaochen Xiao Chen added a comment -

          And please add a release note.
          FYI release note accepts markdowns too, see HADOOP-14003 as an example.

          Show
          xiaochen Xiao Chen added a comment - And please add a release note. FYI release note accepts markdowns too, see HADOOP-14003 as an example.
          Hide
          jzhuge John Zhuge added a comment -

          Thanks Xiao Chen for the great help in review and commit! Thanks Bob Hansen for the comment.

          Show
          jzhuge John Zhuge added a comment - Thanks Xiao Chen for the great help in review and commit! Thanks Bob Hansen for the comment.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11218 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11218/)
          HDFS-10860. Switch HttpFS from Tomcat to Jetty. Contributed by John (xiao: rev 69b23632c48297ac844c58fd3e21aad10c093cc8)

          • (edit) hadoop-assemblies/src/main/resources/assemblies/hadoop-httpfs-dist.xml
          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/CredentialProviderAPI.md
          • (add) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServerWebServer.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/shellprofile.d/hadoop-httpfs.sh
          • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/markdown/index.md
          • (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/logging.properties
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
          • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/conf/httpfs-env.sh
          • (add) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/resources/webapps/static/index.html
          • (add) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServerWebServer.java
          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md
          • (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/server.xml
          • (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/ssl-server.xml.conf
          • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/resources/httpfs-default.xml
          • (add) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/resources/webapps/webhdfs/WEB-INF/web.xml
          • (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh
          • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
          • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh
          • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/servlet/MDCFilter.java
          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/SecureMode.md
          • (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/ROOT/index.html
          • (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/ROOT/WEB-INF/web.xml
          • (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/markdown/ServerSetup.md.vm
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11218 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11218/ ) HDFS-10860 . Switch HttpFS from Tomcat to Jetty. Contributed by John (xiao: rev 69b23632c48297ac844c58fd3e21aad10c093cc8) (edit) hadoop-assemblies/src/main/resources/assemblies/hadoop-httpfs-dist.xml (edit) hadoop-common-project/hadoop-common/src/site/markdown/CredentialProviderAPI.md (add) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServerWebServer.java (add) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/shellprofile.d/hadoop-httpfs.sh (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/markdown/index.md (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/logging.properties (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/conf/httpfs-env.sh (add) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/resources/webapps/static/index.html (add) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServerWebServer.java (edit) hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/server.xml (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/ssl-server.xml.conf (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/resources/httpfs-default.xml (add) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/resources/webapps/webhdfs/WEB-INF/web.xml (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSAuthenticationFilter.java (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/sbin/httpfs.sh (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/servlet/MDCFilter.java (edit) hadoop-common-project/hadoop-common/src/site/markdown/SecureMode.md (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/ROOT/index.html (delete) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/ROOT/WEB-INF/web.xml (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/markdown/ServerSetup.md.vm

            People

            • Assignee:
              jzhuge John Zhuge
              Reporter:
              jzhuge John Zhuge
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development