Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10423

Increase default value of httpfs maxHttpHeaderSize

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.6.4, 3.0.0-alpha1
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: hdfs
    • Labels:
      None

      Description

      The Tomcat default value of maxHttpHeaderSize is 8k, which is too low for certain Hadoop workloads in kerberos enabled environments. This JIRA will to change it to 65536 in server.xml

      1. HDFS-10423.branch-2.patch
        4 kB
        Xiao Chen
      2. HDFS-10423.02.patch
        4 kB
        Nicolae Popa
      3. HDFS-10423.01.patch
        4 kB
        Nicolae Popa
      4. testing-after-HDFS-10423_withCustomHeader44444.txt
        339 kB
        Nicolae Popa
      5. testing-after-HDFS-10423.txt
        340 kB
        Nicolae Popa
      6. testing-before-HDFS-10423.txt
        338 kB
        Nicolae Popa

        Issue Links

          Activity

          Hide
          xiaochen Xiao Chen added a comment -

          Committed to branch-2. Thanks everyone for the contribution.

          Show
          xiaochen Xiao Chen added a comment - Committed to branch-2. Thanks everyone for the contribution.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          0 shelldocs 0m 7s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 41s branch-2 passed
          +1 compile 0m 20s branch-2 passed with JDK v1.8.0_101
          +1 compile 0m 20s branch-2 passed with JDK v1.7.0_111
          +1 mvnsite 0m 46s branch-2 passed
          +1 mvneclipse 0m 16s branch-2 passed
          +1 javadoc 0m 16s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 0m 17s branch-2 passed with JDK v1.7.0_111
          +1 mvninstall 0m 20s the patch passed
          +1 compile 0m 14s the patch passed with JDK v1.8.0_101
          +1 javac 0m 14s the patch passed
          +1 compile 0m 17s the patch passed with JDK v1.7.0_111
          +1 javac 0m 17s the patch passed
          +1 mvnsite 0m 37s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 shellcheck 0m 7s There were no new shellcheck issues.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 0m 11s the patch passed with JDK v1.8.0_101
          +1 javadoc 0m 13s the patch passed with JDK v1.7.0_111
          +1 unit 3m 40s hadoop-hdfs-httpfs in the patch passed with JDK v1.7.0_111.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          20m 44s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HDFS-10423
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834464/HDFS-10423.branch-2.patch
          Optional Tests asflicense mvnsite unit shellcheck shelldocs compile javac javadoc mvninstall xml
          uname Linux c47ccad9ccdd 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / ab36519
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          shellcheck v0.4.4
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17237/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project/hadoop-hdfs-httpfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17237/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. 0 shelldocs 0m 7s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 41s branch-2 passed +1 compile 0m 20s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 20s branch-2 passed with JDK v1.7.0_111 +1 mvnsite 0m 46s branch-2 passed +1 mvneclipse 0m 16s branch-2 passed +1 javadoc 0m 16s branch-2 passed with JDK v1.8.0_101 +1 javadoc 0m 17s branch-2 passed with JDK v1.7.0_111 +1 mvninstall 0m 20s the patch passed +1 compile 0m 14s the patch passed with JDK v1.8.0_101 +1 javac 0m 14s the patch passed +1 compile 0m 17s the patch passed with JDK v1.7.0_111 +1 javac 0m 17s the patch passed +1 mvnsite 0m 37s the patch passed +1 mvneclipse 0m 12s the patch passed +1 shellcheck 0m 7s There were no new shellcheck issues. +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 11s the patch passed with JDK v1.8.0_101 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_111 +1 unit 3m 40s hadoop-hdfs-httpfs in the patch passed with JDK v1.7.0_111. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 20m 44s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-10423 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834464/HDFS-10423.branch-2.patch Optional Tests asflicense mvnsite unit shellcheck shelldocs compile javac javadoc mvninstall xml uname Linux c47ccad9ccdd 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / ab36519 Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 shellcheck v0.4.4 JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17237/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project/hadoop-hdfs-httpfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17237/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          0 shelldocs 0m 10s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          -1 mvninstall 3m 26s root in branch-2 failed.
          +1 compile 0m 21s branch-2 passed with JDK v1.8.0_101
          +1 compile 0m 19s branch-2 passed with JDK v1.7.0_111
          +1 mvnsite 0m 43s branch-2 passed
          +1 mvneclipse 0m 14s branch-2 passed
          +1 javadoc 0m 14s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_111
          +1 mvninstall 0m 21s the patch passed
          +1 compile 0m 15s the patch passed with JDK v1.8.0_101
          +1 javac 0m 15s the patch passed
          +1 compile 0m 17s the patch passed with JDK v1.7.0_111
          +1 javac 0m 17s the patch passed
          +1 mvnsite 0m 39s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 shellcheck 0m 9s There were no new shellcheck issues.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 0m 12s the patch passed with JDK v1.8.0_101
          +1 javadoc 0m 14s the patch passed with JDK v1.7.0_111
          +1 unit 3m 41s hadoop-hdfs-httpfs in the patch passed with JDK v1.7.0_111.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          18m 9s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HDFS-10423
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834464/HDFS-10423.branch-2.patch
          Optional Tests asflicense mvnsite unit shellcheck shelldocs compile javac javadoc mvninstall xml
          uname Linux af4353419799 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / bf6379f
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/17236/artifact/patchprocess/branch-mvninstall-root.txt
          shellcheck v0.4.4
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17236/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project/hadoop-hdfs-httpfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17236/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. 0 shelldocs 0m 10s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 mvninstall 3m 26s root in branch-2 failed. +1 compile 0m 21s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 19s branch-2 passed with JDK v1.7.0_111 +1 mvnsite 0m 43s branch-2 passed +1 mvneclipse 0m 14s branch-2 passed +1 javadoc 0m 14s branch-2 passed with JDK v1.8.0_101 +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_111 +1 mvninstall 0m 21s the patch passed +1 compile 0m 15s the patch passed with JDK v1.8.0_101 +1 javac 0m 15s the patch passed +1 compile 0m 17s the patch passed with JDK v1.7.0_111 +1 javac 0m 17s the patch passed +1 mvnsite 0m 39s the patch passed +1 mvneclipse 0m 12s the patch passed +1 shellcheck 0m 9s There were no new shellcheck issues. +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 12s the patch passed with JDK v1.8.0_101 +1 javadoc 0m 14s the patch passed with JDK v1.7.0_111 +1 unit 3m 41s hadoop-hdfs-httpfs in the patch passed with JDK v1.7.0_111. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 18m 9s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-10423 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834464/HDFS-10423.branch-2.patch Optional Tests asflicense mvnsite unit shellcheck shelldocs compile javac javadoc mvninstall xml uname Linux af4353419799 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / bf6379f Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/17236/artifact/patchprocess/branch-mvninstall-root.txt shellcheck v0.4.4 JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17236/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project/hadoop-hdfs-httpfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17236/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Sorry, reopening to run a pre-commit. Attaching a branch-2 patch.

          Show
          xiaochen Xiao Chen added a comment - Sorry, reopening to run a pre-commit. Attaching a branch-2 patch.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9988 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9988/)
          HDFS-10423. Increase default value of httpfs maxHttpHeaderSize. (atm: rev aa1b583cf99d1a7cfe554d1769fc4af252374663)

          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/conf/httpfs-env.sh
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/markdown/ServerSetup.md.vm
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/server.xml
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/ssl-server.xml.conf
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9988 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9988/ ) HDFS-10423 . Increase default value of httpfs maxHttpHeaderSize. (atm: rev aa1b583cf99d1a7cfe554d1769fc4af252374663) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/conf/httpfs-env.sh hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/markdown/ServerSetup.md.vm hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/server.xml hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/libexec/httpfs-config.sh hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/tomcat/ssl-server.xml.conf
          Hide
          atm Aaron T. Myers added a comment -

          I've just committed this change to trunk.

          Thanks again for the contribution, Nicolae Popa.

          Show
          atm Aaron T. Myers added a comment - I've just committed this change to trunk. Thanks again for the contribution, Nicolae Popa .
          Hide
          atm Aaron T. Myers added a comment -

          The latest patch looks good to me, and thanks a lot Nicolae Popa for describing the manual testing you did.

          +1, I'm going to commit this momentarily.

          Show
          atm Aaron T. Myers added a comment - The latest patch looks good to me, and thanks a lot Nicolae Popa for describing the manual testing you did. +1, I'm going to commit this momentarily.
          Hide
          npopa Nicolae Popa added a comment -

          Patch was manually tested locally and passed. See the attached notes.
          I generated http requests of various sizes (1024 2048 4096 8192 8193 16384 32768 44443 44444 44445 65536 65537) in bytes and observed the results in the following scenarios:
          1) before the patch
          2) after the patch
          3) with the export HTTPFS_MAX_HTTP_HEADER_SIZE=44444 in etc/hadoop/httpfs-env.sh

          In testing-before-HTTP-10423.txt we can see it starts failing after a HTTP 8192 request
          In testing-after-HTTP-10423.txt we can see it starts failing after a HTTP 65536 request since this is the new default (as expected)
          I also run a test setting export HTTPFS_MAX_HTTP_HEADER_SIZE=44444 in etc/hadoop/httpfs-env.sh and the behavior was as expected. It worked with an HTTP requests up to 44444 bytes and failed with a request of 44445 bytes (as expected). (see testing-after-HDFS-10423_withCustomHeader44444.txt)

          Show
          npopa Nicolae Popa added a comment - Patch was manually tested locally and passed. See the attached notes. I generated http requests of various sizes (1024 2048 4096 8192 8193 16384 32768 44443 44444 44445 65536 65537) in bytes and observed the results in the following scenarios: 1) before the patch 2) after the patch 3) with the export HTTPFS_MAX_HTTP_HEADER_SIZE=44444 in etc/hadoop/httpfs-env.sh In testing-before-HTTP-10423.txt we can see it starts failing after a HTTP 8192 request In testing-after-HTTP-10423.txt we can see it starts failing after a HTTP 65536 request since this is the new default (as expected) I also run a test setting export HTTPFS_MAX_HTTP_HEADER_SIZE=44444 in etc/hadoop/httpfs-env.sh and the behavior was as expected. It worked with an HTTP requests up to 44444 bytes and failed with a request of 44445 bytes (as expected). (see testing-after- HDFS-10423 _withCustomHeader44444.txt)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 31s Docker mode activated.
          0 shelldocs 0m 4s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 21s trunk passed
          +1 compile 0m 18s trunk passed
          +1 mvnsite 0m 30s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 javadoc 0m 20s trunk passed
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 16s the patch passed
          +1 javac 0m 16s the patch passed
          +1 mvnsite 0m 23s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 shellcheck 0m 14s The patch generated 0 new + 80 unchanged - 1 fixed = 80 total (was 81)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 0m 13s the patch passed
          +1 unit 3m 28s hadoop-hdfs-httpfs in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          17m 1s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808949/HDFS-10423.02.patch
          JIRA Issue HDFS-10423
          Optional Tests asflicense mvnsite unit shellcheck shelldocs compile javac javadoc mvninstall xml
          uname Linux d5a5d94dd1b6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8c8a377
          Default Java 1.8.0_91
          shellcheck v0.4.4
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15705/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project/hadoop-hdfs-httpfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15705/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 31s Docker mode activated. 0 shelldocs 0m 4s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 21s trunk passed +1 compile 0m 18s trunk passed +1 mvnsite 0m 30s trunk passed +1 mvneclipse 0m 13s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 16s the patch passed +1 javac 0m 16s the patch passed +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 9s the patch passed +1 shellcheck 0m 14s The patch generated 0 new + 80 unchanged - 1 fixed = 80 total (was 81) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 13s the patch passed +1 unit 3m 28s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 17m 1s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808949/HDFS-10423.02.patch JIRA Issue HDFS-10423 Optional Tests asflicense mvnsite unit shellcheck shelldocs compile javac javadoc mvninstall xml uname Linux d5a5d94dd1b6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8c8a377 Default Java 1.8.0_91 shellcheck v0.4.4 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15705/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project/hadoop-hdfs-httpfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15705/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          0 shelldocs 0m 4s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 11s trunk passed
          +1 compile 0m 17s trunk passed
          +1 mvnsite 0m 25s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 javadoc 0m 17s trunk passed
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 18s the patch passed
          +1 javac 0m 18s the patch passed
          +1 mvnsite 0m 24s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 shellcheck 0m 12s The patch generated 0 new + 80 unchanged - 1 fixed = 80 total (was 81)
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 0m 12s the patch passed
          +1 unit 3m 17s hadoop-hdfs-httpfs in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          14m 12s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808944/HDFS-10423.01.patch
          JIRA Issue HDFS-10423
          Optional Tests asflicense mvnsite unit shellcheck shelldocs compile javac javadoc mvninstall xml
          uname Linux 7bd92a38ec4e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8c8a377
          Default Java 1.8.0_91
          shellcheck v0.4.4
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15704/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project/hadoop-hdfs-httpfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15704/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. 0 shelldocs 0m 4s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 11s trunk passed +1 compile 0m 17s trunk passed +1 mvnsite 0m 25s trunk passed +1 mvneclipse 0m 12s trunk passed +1 javadoc 0m 17s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 18s the patch passed +1 javac 0m 18s the patch passed +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 9s the patch passed +1 shellcheck 0m 12s The patch generated 0 new + 80 unchanged - 1 fixed = 80 total (was 81) +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 12s the patch passed +1 unit 3m 17s hadoop-hdfs-httpfs in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 14m 12s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808944/HDFS-10423.01.patch JIRA Issue HDFS-10423 Optional Tests asflicense mvnsite unit shellcheck shelldocs compile javac javadoc mvninstall xml uname Linux 7bd92a38ec4e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8c8a377 Default Java 1.8.0_91 shellcheck v0.4.4 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15704/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project/hadoop-hdfs-httpfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15704/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          npopa Nicolae Popa added a comment -

          Re-attaching the same patch with an increase version and resubmit.

          Show
          npopa Nicolae Popa added a comment - Re-attaching the same patch with an increase version and resubmit.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 0m 6s Docker failed to build yetus/hadoop:2c91fd8.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808592/HDFS-10423.01.patch
          JIRA Issue HDFS-10423
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15702/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 6s Docker failed to build yetus/hadoop:2c91fd8. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808592/HDFS-10423.01.patch JIRA Issue HDFS-10423 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15702/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          npopa Nicolae Popa added a comment -

          I have attached the tested patch. The test is using the below script to generate a bloat header for http using the below ad hoc script:
          It will generate an http request with various sizes to stress the tomcat settings.

          for size in 1024 2048 4096 8192 8193 16384 32768 44443 44444 44445 65536 65537 ; do
          echo "################################################################################################################################################################"
          echo "### Trying with request size: $size"
          echo "################################################################################################################################################################"
          echo
          http_request_size=$size
          cat <<-EOF>http_request
          GET /webhdfs/v1/user/npopa?op=liststatus&user.name=npopa HTTP/1.1
          Host: localhost:14000
          EOF
          http_request_size_tmp=`wc -c http_request|awk '

          {print $1}

          '`
          export to_fill=`expr $http_request_size - $http_request_size_tmp - 5` # -5 comes from "h: " and new lines before and after the header
          export RNDSTR=`cat /dev/urandom | env LC_CTYPE=C tr -dc 'a-zA-Z0-9' | fold -w $to_fill| head -n 1`
          echo "h: $RNDSTR">>http_request
          echo "">>http_request
          echo "Request is: "
          cat http_request
          echo
          echo "Request size is: "
          ls -ltr http_request
          echo
          echo "Result is: "
          nc localhost 14000 < http_request
          echo
          done

          The scenarios and results are:
          1) before the patch
          2) after the patch
          3) with the export HTTPFS_MAX_HTTP_HEADER_SIZE=44444 in etc/hadoop/httpfs-env.sh

          In before-HTTP-10423.txt we can see it starts failing after a HTTP 8192 request
          In after-HTTP-10423.txt we can see it starts failing after a HTTP 65536 request (as expected)
          I also run a test setting export HTTPFS_MAX_HTTP_HEADER_SIZE=44444 in etc/hadoop/httpfs-env.sh and the behavior was as expected. It worked with an HTTP request of 44444 bytes and failed with a request of 44445 bytes (as expected). (see after-HDFS-10423_withCustomHeader44444.txt)

          For each size below are the expected results:

          successful result is:
          Result is:
          HTTP/1.1 200 OK

          failed result is:
          Result is:
          HTTP/1.1 400 Bad Request

          Show
          npopa Nicolae Popa added a comment - I have attached the tested patch. The test is using the below script to generate a bloat header for http using the below ad hoc script: It will generate an http request with various sizes to stress the tomcat settings. for size in 1024 2048 4096 8192 8193 16384 32768 44443 44444 44445 65536 65537 ; do echo "################################################################################################################################################################" echo "### Trying with request size: $size" echo "################################################################################################################################################################" echo http_request_size=$size cat <<-EOF>http_request GET /webhdfs/v1/user/npopa?op=liststatus&user.name=npopa HTTP/1.1 Host: localhost:14000 EOF http_request_size_tmp=`wc -c http_request|awk ' {print $1} '` export to_fill=`expr $http_request_size - $http_request_size_tmp - 5` # -5 comes from "h: " and new lines before and after the header export RNDSTR=`cat /dev/urandom | env LC_CTYPE=C tr -dc 'a-zA-Z0-9' | fold -w $to_fill| head -n 1` echo "h: $RNDSTR">>http_request echo "">>http_request echo "Request is: " cat http_request echo echo "Request size is: " ls -ltr http_request echo echo "Result is: " nc localhost 14000 < http_request echo done The scenarios and results are: 1) before the patch 2) after the patch 3) with the export HTTPFS_MAX_HTTP_HEADER_SIZE=44444 in etc/hadoop/httpfs-env.sh In before-HTTP-10423.txt we can see it starts failing after a HTTP 8192 request In after-HTTP-10423.txt we can see it starts failing after a HTTP 65536 request (as expected) I also run a test setting export HTTPFS_MAX_HTTP_HEADER_SIZE=44444 in etc/hadoop/httpfs-env.sh and the behavior was as expected. It worked with an HTTP request of 44444 bytes and failed with a request of 44445 bytes (as expected). (see after- HDFS-10423 _withCustomHeader44444.txt) For each size below are the expected results: successful result is: Result is: HTTP/1.1 200 OK failed result is: Result is: HTTP/1.1 400 Bad Request
          Hide
          xyao Xiaoyu Yao added a comment - - edited

          Nicolae Popa, thanks for reporting the issue. This can be done similar to HADOOP-12764 for kms. There are some problem assigning the JIRA to you.
          Below is the error message I got: "User 'Nicolae Popa' does not exist."
          There are some JIRA permission issues lately. Feel free to post your patch and we will fix permission issue separately.

          Show
          xyao Xiaoyu Yao added a comment - - edited Nicolae Popa , thanks for reporting the issue. This can be done similar to HADOOP-12764 for kms. There are some problem assigning the JIRA to you. Below is the error message I got: "User 'Nicolae Popa' does not exist." There are some JIRA permission issues lately. Feel free to post your patch and we will fix permission issue separately.
          Hide
          npopa Nicolae Popa added a comment -

          How do I assign this to myself?

          Show
          npopa Nicolae Popa added a comment - How do I assign this to myself?

            People

            • Assignee:
              npopa Nicolae Popa
              Reporter:
              npopa Nicolae Popa
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development