Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10720

YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging

Details

    • Reviewed

    Description

      Following is proxy server show, too many connections from one client, this caused the proxy server hang, and the yarn web can't jump to web proxy.

      Following is the AM which is abnormal, but proxy server don't know it is abnormal already, so the connections can't be closed, we should add time out support in proxy server to prevent this. And one abnormal AM may cause hundreds even thousands of connections, it is very heavy.

       

      After i kill the abnormal AM, the proxy server become healthy. This case happened many times in our production clusters, our clusters are huge, and the abnormal AM will be existed in a regular case.

       

      I will add timeout supported in web proxy server in this jira.

       

      cc  pbacsko ebadger Jim_Brennan  ztang  epayne gandras  bteke

       

      Attachments

        1. YARN-10720.006.patch
          11 kB
          Qi Zhu
        2. YARN-10720.005.patch
          11 kB
          Qi Zhu
        3. YARN-10720.004.patch
          11 kB
          Qi Zhu
        4. YARN-10720.003.patch
          12 kB
          Qi Zhu
        5. YARN-10720.002.patch
          12 kB
          Qi Zhu
        6. YARN-10720.001.patch
          11 kB
          Qi Zhu
        7. image-2021-03-29-14-05-32-708.png
          182 kB
          Qi Zhu
        8. image-2021-03-29-14-04-33-776.png
          95 kB
          Qi Zhu

        Issue Links

          Activity

            zhuqi Qi Zhu added a comment - - edited

            cc  pbacsko ebadger Jim_Brennan  ztang  epayne gandras  bteke brahmareddy

             

            Could you help review this?

            Tested in our test cluster, works well.

            Thanks.

            zhuqi Qi Zhu added a comment - - edited cc   pbacsko   ebadger   Jim_Brennan    ztang    epayne   gandras    bteke brahmareddy   Could you help review this? Tested in our test cluster, works well. Thanks.
            hadoopci Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Logfile Comment
            0 reexec 1m 51s Docker mode activated.
                  Prechecks  
            +1 dupname 0m 0s No case conflicting files found.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 0m 0s test4tests The patch appears to include 1 new or modified test files.
                  trunk Compile Tests  
            0 mvndep 2m 4s Maven dependency ordering for branch
            +1 mvninstall 24m 23s trunk passed
            +1 compile 10m 41s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 compile 9m 8s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            +1 checkstyle 1m 42s trunk passed
            +1 mvnsite 1m 19s trunk passed
            +1 shadedclient 18m 21s branch has no errors when building and testing our client artifacts.
            +1 javadoc 1m 12s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 javadoc 1m 5s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            0 spotbugs 23m 20s Both FindBugs and SpotBugs are enabled, using SpotBugs.
            +1 spotbugs 2m 44s trunk passed
                  Patch Compile Tests  
            0 mvndep 0m 21s Maven dependency ordering for patch
            +1 mvninstall 0m 55s the patch passed
            +1 compile 9m 10s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            -1 javac 9m 10s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/862/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 1 new + 385 unchanged - 0 fixed = 386 total (was 385)
            +1 compile 8m 21s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            -1 javac 8m 21s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/862/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 1 new + 354 unchanged - 0 fixed = 355 total (was 354)
            +1 checkstyle 1m 40s the patch passed
            +1 mvnsite 1m 14s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 shadedclient 14m 58s patch has no errors when building and testing our client artifacts.
            +1 javadoc 1m 7s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 javadoc 1m 4s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            +1 spotbugs 3m 9s the patch passed
                  Other Tests  
            +1 unit 0m 56s hadoop-yarn-api in the patch passed.
            +1 unit 1m 14s hadoop-yarn-server-web-proxy in the patch passed.
            +1 asflicense 0m 41s The patch does not generate ASF License warnings.
            119m 25s



            Subsystem Report/Notes
            Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/862/artifact/out/Dockerfile
            JIRA Issue YARN-10720
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13023101/YARN-10720.001.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs
            uname Linux 4bfe826d84a3 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality personality/hadoop.sh
            git revision trunk / ea6595d3b68
            Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/862/testReport/
            Max. process+thread count 516 (vs. ulimit of 5500)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn
            Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/862/console
            versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
            Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

            This message was automatically generated.

            hadoopci Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Logfile Comment 0 reexec 1m 51s Docker mode activated.       Prechecks   +1 dupname 0m 0s No case conflicting files found. +1 @author 0m 0s The patch does not contain any @author tags. +1 0m 0s test4tests The patch appears to include 1 new or modified test files.       trunk Compile Tests   0 mvndep 2m 4s Maven dependency ordering for branch +1 mvninstall 24m 23s trunk passed +1 compile 10m 41s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 compile 9m 8s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 +1 checkstyle 1m 42s trunk passed +1 mvnsite 1m 19s trunk passed +1 shadedclient 18m 21s branch has no errors when building and testing our client artifacts. +1 javadoc 1m 12s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 javadoc 1m 5s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 0 spotbugs 23m 20s Both FindBugs and SpotBugs are enabled, using SpotBugs. +1 spotbugs 2m 44s trunk passed       Patch Compile Tests   0 mvndep 0m 21s Maven dependency ordering for patch +1 mvninstall 0m 55s the patch passed +1 compile 9m 10s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 -1 javac 9m 10s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/862/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 1 new + 385 unchanged - 0 fixed = 386 total (was 385) +1 compile 8m 21s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 -1 javac 8m 21s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/862/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 1 new + 354 unchanged - 0 fixed = 355 total (was 354) +1 checkstyle 1m 40s the patch passed +1 mvnsite 1m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 14m 58s patch has no errors when building and testing our client artifacts. +1 javadoc 1m 7s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 javadoc 1m 4s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 +1 spotbugs 3m 9s the patch passed       Other Tests   +1 unit 0m 56s hadoop-yarn-api in the patch passed. +1 unit 1m 14s hadoop-yarn-server-web-proxy in the patch passed. +1 asflicense 0m 41s The patch does not generate ASF License warnings. 119m 25s Subsystem Report/Notes Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/862/artifact/out/Dockerfile JIRA Issue YARN-10720 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13023101/YARN-10720.001.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs uname Linux 4bfe826d84a3 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality personality/hadoop.sh git revision trunk / ea6595d3b68 Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/862/testReport/ Max. process+thread count 516 (vs. ulimit of 5500) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/862/console versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org This message was automatically generated.
            hadoopci Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Logfile Comment
            0 reexec 0m 44s Docker mode activated.
                  Prechecks  
            +1 dupname 0m 0s No case conflicting files found.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 0m 0s test4tests The patch appears to include 1 new or modified test files.
                  trunk Compile Tests  
            0 mvndep 1m 48s Maven dependency ordering for branch
            +1 mvninstall 20m 45s trunk passed
            +1 compile 9m 30s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 compile 8m 36s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            +1 checkstyle 1m 40s trunk passed
            +1 mvnsite 2m 40s trunk passed
            +1 shadedclient 17m 56s branch has no errors when building and testing our client artifacts.
            +1 javadoc 2m 19s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 javadoc 2m 30s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            0 spotbugs 27m 35s Both FindBugs and SpotBugs are enabled, using SpotBugs.
            +1 spotbugs 4m 52s trunk passed
                  Patch Compile Tests  
            0 mvndep 0m 26s Maven dependency ordering for patch
            +1 mvninstall 1m 39s the patch passed
            +1 compile 8m 30s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            -1 javac 8m 30s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 1 new + 384 unchanged - 0 fixed = 385 total (was 384)
            +1 compile 7m 49s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            -1 javac 7m 49s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 1 new + 355 unchanged - 0 fixed = 356 total (was 355)
            +1 checkstyle 1m 39s the patch passed
            +1 mvnsite 2m 23s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 2s The patch has no ill-formed XML file.
            +1 shadedclient 13m 3s patch has no errors when building and testing our client artifacts.
            +1 javadoc 2m 10s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 javadoc 2m 21s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            +1 spotbugs 5m 21s the patch passed
                  Other Tests  
            +1 unit 1m 5s hadoop-yarn-api in the patch passed.
            +1 unit 4m 53s hadoop-yarn-common in the patch passed.
            -1 unit 1m 18s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt hadoop-yarn-server-web-proxy in the patch passed.
            +1 asflicense 0m 53s The patch does not generate ASF License warnings.
            126m 34s



            Reason Tests
            Failed junit tests hadoop.yarn.server.webproxy.TestWebAppProxyServlet



            Subsystem Report/Notes
            Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/artifact/out/Dockerfile
            JIRA Issue YARN-10720
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13023102/YARN-10720.002.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs xml
            uname Linux 1bd152c39c2d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality personality/hadoop.sh
            git revision trunk / ea6595d3b68
            Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/testReport/
            Max. process+thread count 716 (vs. ulimit of 5500)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn
            Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/console
            versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
            Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

            This message was automatically generated.

            hadoopci Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Logfile Comment 0 reexec 0m 44s Docker mode activated.       Prechecks   +1 dupname 0m 0s No case conflicting files found. +1 @author 0m 0s The patch does not contain any @author tags. +1 0m 0s test4tests The patch appears to include 1 new or modified test files.       trunk Compile Tests   0 mvndep 1m 48s Maven dependency ordering for branch +1 mvninstall 20m 45s trunk passed +1 compile 9m 30s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 compile 8m 36s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 +1 checkstyle 1m 40s trunk passed +1 mvnsite 2m 40s trunk passed +1 shadedclient 17m 56s branch has no errors when building and testing our client artifacts. +1 javadoc 2m 19s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 javadoc 2m 30s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 0 spotbugs 27m 35s Both FindBugs and SpotBugs are enabled, using SpotBugs. +1 spotbugs 4m 52s trunk passed       Patch Compile Tests   0 mvndep 0m 26s Maven dependency ordering for patch +1 mvninstall 1m 39s the patch passed +1 compile 8m 30s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 -1 javac 8m 30s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 1 new + 384 unchanged - 0 fixed = 385 total (was 384) +1 compile 7m 49s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 -1 javac 7m 49s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 1 new + 355 unchanged - 0 fixed = 356 total (was 355) +1 checkstyle 1m 39s the patch passed +1 mvnsite 2m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 shadedclient 13m 3s patch has no errors when building and testing our client artifacts. +1 javadoc 2m 10s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 javadoc 2m 21s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 +1 spotbugs 5m 21s the patch passed       Other Tests   +1 unit 1m 5s hadoop-yarn-api in the patch passed. +1 unit 4m 53s hadoop-yarn-common in the patch passed. -1 unit 1m 18s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt hadoop-yarn-server-web-proxy in the patch passed. +1 asflicense 0m 53s The patch does not generate ASF License warnings. 126m 34s Reason Tests Failed junit tests hadoop.yarn.server.webproxy.TestWebAppProxyServlet Subsystem Report/Notes Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/artifact/out/Dockerfile JIRA Issue YARN-10720 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13023102/YARN-10720.002.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs xml uname Linux 1bd152c39c2d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality personality/hadoop.sh git revision trunk / ea6595d3b68 Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/testReport/ Max. process+thread count 716 (vs. ulimit of 5500) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/console versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org This message was automatically generated.
            hadoopci Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Logfile Comment
            0 reexec 1m 15s Docker mode activated.
                  Prechecks  
            +1 dupname 0m 0s No case conflicting files found.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 0m 0s test4tests The patch appears to include 1 new or modified test files.
                  trunk Compile Tests  
            0 mvndep 0m 20s Maven dependency ordering for branch
            +1 mvninstall 21m 37s trunk passed
            +1 compile 9m 41s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 compile 11m 8s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            +1 checkstyle 2m 0s trunk passed
            +1 mvnsite 2m 43s trunk passed
            +1 shadedclient 22m 12s branch has no errors when building and testing our client artifacts.
            +1 javadoc 1m 54s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 javadoc 2m 6s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            0 spotbugs 30m 52s Both FindBugs and SpotBugs are enabled, using SpotBugs.
            +1 spotbugs 4m 43s trunk passed
                  Patch Compile Tests  
            0 mvndep 0m 20s Maven dependency ordering for patch
            +1 mvninstall 1m 34s the patch passed
            +1 compile 9m 15s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            -1 javac 9m 15s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/866/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 1 new + 384 unchanged - 0 fixed = 385 total (was 384)
            +1 compile 8m 18s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            -1 javac 8m 18s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/866/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 1 new + 354 unchanged - 0 fixed = 355 total (was 354)
            +1 checkstyle 1m 38s the patch passed
            +1 mvnsite 2m 6s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 shadedclient 15m 8s patch has no errors when building and testing our client artifacts.
            +1 javadoc 1m 51s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 javadoc 2m 2s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            +1 spotbugs 5m 14s the patch passed
                  Other Tests  
            +1 unit 0m 56s hadoop-yarn-api in the patch passed.
            +1 unit 4m 38s hadoop-yarn-common in the patch passed.
            +1 unit 1m 12s hadoop-yarn-server-web-proxy in the patch passed.
            +1 asflicense 0m 42s The patch does not generate ASF License warnings.
            133m 10s



            Subsystem Report/Notes
            Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/866/artifact/out/Dockerfile
            JIRA Issue YARN-10720
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13023121/YARN-10720.003.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs xml
            uname Linux fd01596f6811 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality personality/hadoop.sh
            git revision trunk / ea6595d3b68
            Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/866/testReport/
            Max. process+thread count 515 (vs. ulimit of 5500)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn
            Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/866/console
            versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
            Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

            This message was automatically generated.

            hadoopci Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Logfile Comment 0 reexec 1m 15s Docker mode activated.       Prechecks   +1 dupname 0m 0s No case conflicting files found. +1 @author 0m 0s The patch does not contain any @author tags. +1 0m 0s test4tests The patch appears to include 1 new or modified test files.       trunk Compile Tests   0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 21m 37s trunk passed +1 compile 9m 41s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 compile 11m 8s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 +1 checkstyle 2m 0s trunk passed +1 mvnsite 2m 43s trunk passed +1 shadedclient 22m 12s branch has no errors when building and testing our client artifacts. +1 javadoc 1m 54s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 javadoc 2m 6s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 0 spotbugs 30m 52s Both FindBugs and SpotBugs are enabled, using SpotBugs. +1 spotbugs 4m 43s trunk passed       Patch Compile Tests   0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 1m 34s the patch passed +1 compile 9m 15s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 -1 javac 9m 15s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/866/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 1 new + 384 unchanged - 0 fixed = 385 total (was 384) +1 compile 8m 18s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 -1 javac 8m 18s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/866/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 1 new + 354 unchanged - 0 fixed = 355 total (was 354) +1 checkstyle 1m 38s the patch passed +1 mvnsite 2m 6s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 shadedclient 15m 8s patch has no errors when building and testing our client artifacts. +1 javadoc 1m 51s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 javadoc 2m 2s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 +1 spotbugs 5m 14s the patch passed       Other Tests   +1 unit 0m 56s hadoop-yarn-api in the patch passed. +1 unit 4m 38s hadoop-yarn-common in the patch passed. +1 unit 1m 12s hadoop-yarn-server-web-proxy in the patch passed. +1 asflicense 0m 42s The patch does not generate ASF License warnings. 133m 10s Subsystem Report/Notes Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/866/artifact/out/Dockerfile JIRA Issue YARN-10720 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13023121/YARN-10720.003.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs xml uname Linux fd01596f6811 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality personality/hadoop.sh git revision trunk / ea6595d3b68 Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/866/testReport/ Max. process+thread count 515 (vs. ulimit of 5500) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/866/console versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org This message was automatically generated.
            zhuqi Qi Zhu added a comment - - edited

            Hi pbacsko gandras 

            Could you help review this?

            It will be a great improvement for the regular proxy server hang problem in our big production clusters, and without this, we should restart the hang web proxy server regularly.

            And, i have a question:

            If the none() in ExpectedException has been deprecated should be fixed in java doc here.

            Thanks.

            zhuqi Qi Zhu added a comment - - edited Hi  pbacsko   gandras   Could you help review this? It will be a great improvement for the regular proxy server hang problem in our big production clusters, and without this, we should restart the hang web proxy server regularly. And, i have a question: If the none() in ExpectedException has been deprecated should be fixed in java doc here. Thanks.
            pbacsko Peter Bacsko added a comment -

            Thanks zhuqi for the patch.

            1. As you said ExpectedException.none() has been deprecated. Either use the new assertThrows() or @Test(expected = SocketTimeoutException.class), I think using the second is easier.

            2.

            	    conf.setInt(YarnConfiguration.RM_PROXY_CONNECTION_TIMEOUT,
            	        1 * 1000);
            

            Just write "1000" instead of "1 * 1000".

            3.

            	    try {
            	      when(response.getOutputStream()).thenReturn(null);
            	    } catch (IOException e) {
            	      e.printStackTrace();
            	    }
            

            Unnecessary try-catch block. The method already has a throws clause.

            4.

            	    @Override
            	    protected void doGet(HttpServletRequest req, HttpServletResponse resp)
            	        throws ServletException, IOException {
            	      try {
            	        Thread.sleep(10 * 1000);
            	      } catch (InterruptedException e) {
            	        e.printStackTrace();
            	      }
            	      resp.setStatus(HttpServletResponse.SC_OK);
            	    }
            

            Maybe a minor thing, but if you catch InterruptedException, don't just print the stack trace, log it with LOG.warn("doGet() interrupted", e). In this case, I'd also return with HttpServletResponse.SC_BAD_REQUEST.

            5.
            <description>The web proxy connection timeout, default is 60s(60 * 1000ms).</description>

            This already goes to yarn-default.xml, so you can omit the part "default is 60s(60 * 1000ms)" and just write "The web proxy connection timeout".

            pbacsko Peter Bacsko added a comment - Thanks zhuqi for the patch. 1. As you said ExpectedException.none() has been deprecated. Either use the new assertThrows() or @Test(expected = SocketTimeoutException.class) , I think using the second is easier. 2. conf.setInt(YarnConfiguration.RM_PROXY_CONNECTION_TIMEOUT, 1 * 1000); Just write "1000" instead of "1 * 1000". 3. try { when(response.getOutputStream()).thenReturn(null); } catch (IOException e) { e.printStackTrace(); } Unnecessary try-catch block. The method already has a throws clause. 4. @Override protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException { try { Thread.sleep(10 * 1000); } catch (InterruptedException e) { e.printStackTrace(); } resp.setStatus(HttpServletResponse.SC_OK); } Maybe a minor thing, but if you catch InterruptedException , don't just print the stack trace, log it with LOG.warn("doGet() interrupted", e) . In this case, I'd also return with HttpServletResponse.SC_BAD_REQUEST . 5. <description>The web proxy connection timeout, default is 60s(60 * 1000ms).</description> This already goes to yarn-default.xml , so you can omit the part "default is 60s(60 * 1000ms)" and just write "The web proxy connection timeout".
            zhuqi Qi Zhu added a comment -

            Thanks a lot pbacsko for valid suggestions. 

            I have updated in latest patch.

            zhuqi Qi Zhu added a comment - Thanks a lot pbacsko  for valid suggestions.  I have updated in latest patch.
            hadoopci Hadoop QA added a comment -
            +1 overall



            Vote Subsystem Runtime Logfile Comment
            0 reexec 0m 53s Docker mode activated.
                  Prechecks  
            +1 dupname 0m 0s No case conflicting files found.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 0m 0s test4tests The patch appears to include 1 new or modified test files.
                  trunk Compile Tests  
            0 mvndep 1m 52s Maven dependency ordering for branch
            +1 mvninstall 19m 47s trunk passed
            +1 compile 8m 59s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 compile 7m 49s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            +1 checkstyle 1m 44s trunk passed
            +1 mvnsite 2m 36s trunk passed
            +1 shadedclient 17m 42s branch has no errors when building and testing our client artifacts.
            +1 javadoc 2m 18s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 javadoc 2m 32s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            0 spotbugs 27m 21s Both FindBugs and SpotBugs are enabled, using SpotBugs.
            +1 spotbugs 4m 52s trunk passed
                  Patch Compile Tests  
            0 mvndep 0m 26s Maven dependency ordering for patch
            +1 mvninstall 1m 38s the patch passed
            +1 compile 8m 25s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 javac 8m 25s the patch passed
            +1 compile 7m 48s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            +1 javac 7m 48s the patch passed
            -0 checkstyle 1m 41s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/883/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 258 unchanged - 0 fixed = 260 total (was 258)
            +1 mvnsite 2m 23s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 shadedclient 13m 3s patch has no errors when building and testing our client artifacts.
            +1 javadoc 2m 13s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 javadoc 2m 24s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            +1 spotbugs 5m 21s the patch passed
                  Other Tests  
            +1 unit 1m 4s hadoop-yarn-api in the patch passed.
            +1 unit 4m 54s hadoop-yarn-common in the patch passed.
            +1 unit 1m 20s hadoop-yarn-server-web-proxy in the patch passed.
            +1 asflicense 0m 49s The patch does not generate ASF License warnings.
            124m 15s



            Subsystem Report/Notes
            Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/883/artifact/out/Dockerfile
            JIRA Issue YARN-10720
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13023234/YARN-10720.004.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs xml
            uname Linux 03bbeb7415a2 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality personality/hadoop.sh
            git revision trunk / ff6ec20d845
            Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/883/testReport/
            Max. process+thread count 597 (vs. ulimit of 5500)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn
            Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/883/console
            versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
            Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

            This message was automatically generated.

            hadoopci Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Logfile Comment 0 reexec 0m 53s Docker mode activated.       Prechecks   +1 dupname 0m 0s No case conflicting files found. +1 @author 0m 0s The patch does not contain any @author tags. +1 0m 0s test4tests The patch appears to include 1 new or modified test files.       trunk Compile Tests   0 mvndep 1m 52s Maven dependency ordering for branch +1 mvninstall 19m 47s trunk passed +1 compile 8m 59s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 compile 7m 49s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 +1 checkstyle 1m 44s trunk passed +1 mvnsite 2m 36s trunk passed +1 shadedclient 17m 42s branch has no errors when building and testing our client artifacts. +1 javadoc 2m 18s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 javadoc 2m 32s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 0 spotbugs 27m 21s Both FindBugs and SpotBugs are enabled, using SpotBugs. +1 spotbugs 4m 52s trunk passed       Patch Compile Tests   0 mvndep 0m 26s Maven dependency ordering for patch +1 mvninstall 1m 38s the patch passed +1 compile 8m 25s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 javac 8m 25s the patch passed +1 compile 7m 48s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 +1 javac 7m 48s the patch passed -0 checkstyle 1m 41s https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/883/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 258 unchanged - 0 fixed = 260 total (was 258) +1 mvnsite 2m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 shadedclient 13m 3s patch has no errors when building and testing our client artifacts. +1 javadoc 2m 13s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 javadoc 2m 24s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 +1 spotbugs 5m 21s the patch passed       Other Tests   +1 unit 1m 4s hadoop-yarn-api in the patch passed. +1 unit 4m 54s hadoop-yarn-common in the patch passed. +1 unit 1m 20s hadoop-yarn-server-web-proxy in the patch passed. +1 asflicense 0m 49s The patch does not generate ASF License warnings. 124m 15s Subsystem Report/Notes Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/883/artifact/out/Dockerfile JIRA Issue YARN-10720 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13023234/YARN-10720.004.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs xml uname Linux 03bbeb7415a2 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality personality/hadoop.sh git revision trunk / ff6ec20d845 Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/883/testReport/ Max. process+thread count 597 (vs. ulimit of 5500) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/883/console versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org This message was automatically generated.
            zhuqi Qi Zhu added a comment -

            Removed the unused import in latest patch. 

            zhuqi Qi Zhu added a comment - Removed the unused import in latest patch. 
            hadoopci Hadoop QA added a comment -
            +1 overall



            Vote Subsystem Runtime Logfile Comment
            0 reexec 1m 29s Docker mode activated.
                  Prechecks  
            +1 dupname 0m 0s No case conflicting files found.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 0m 0s test4tests The patch appears to include 1 new or modified test files.
                  trunk Compile Tests  
            0 mvndep 1m 36s Maven dependency ordering for branch
            +1 mvninstall 22m 15s trunk passed
            +1 compile 9m 46s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 compile 8m 14s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            +1 checkstyle 1m 42s trunk passed
            +1 mvnsite 2m 14s trunk passed
            +1 shadedclient 19m 32s branch has no errors when building and testing our client artifacts.
            +1 javadoc 1m 55s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 javadoc 2m 8s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            0 spotbugs 28m 47s Both FindBugs and SpotBugs are enabled, using SpotBugs.
            +1 spotbugs 5m 18s trunk passed
                  Patch Compile Tests  
            0 mvndep 0m 27s Maven dependency ordering for patch
            +1 mvninstall 2m 2s the patch passed
            +1 compile 12m 18s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 javac 12m 18s the patch passed
            +1 compile 9m 30s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            +1 javac 9m 30s the patch passed
            +1 checkstyle 1m 39s the patch passed
            +1 mvnsite 2m 9s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 2s The patch has no ill-formed XML file.
            +1 shadedclient 16m 21s patch has no errors when building and testing our client artifacts.
            +1 javadoc 2m 20s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 javadoc 2m 20s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            +1 spotbugs 5m 43s the patch passed
                  Other Tests  
            +1 unit 1m 5s hadoop-yarn-api in the patch passed.
            +1 unit 5m 1s hadoop-yarn-common in the patch passed.
            +1 unit 1m 12s hadoop-yarn-server-web-proxy in the patch passed.
            +1 asflicense 0m 41s The patch does not generate ASF License warnings.
            138m 46s



            Subsystem Report/Notes
            Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/884/artifact/out/Dockerfile
            JIRA Issue YARN-10720
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13023244/YARN-10720.005.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs xml
            uname Linux bf7e7f627862 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality personality/hadoop.sh
            git revision trunk / ff6ec20d845
            Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/884/testReport/
            Max. process+thread count 567 (vs. ulimit of 5500)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn
            Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/884/console
            versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
            Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

            This message was automatically generated.

            hadoopci Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Logfile Comment 0 reexec 1m 29s Docker mode activated.       Prechecks   +1 dupname 0m 0s No case conflicting files found. +1 @author 0m 0s The patch does not contain any @author tags. +1 0m 0s test4tests The patch appears to include 1 new or modified test files.       trunk Compile Tests   0 mvndep 1m 36s Maven dependency ordering for branch +1 mvninstall 22m 15s trunk passed +1 compile 9m 46s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 compile 8m 14s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 +1 checkstyle 1m 42s trunk passed +1 mvnsite 2m 14s trunk passed +1 shadedclient 19m 32s branch has no errors when building and testing our client artifacts. +1 javadoc 1m 55s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 javadoc 2m 8s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 0 spotbugs 28m 47s Both FindBugs and SpotBugs are enabled, using SpotBugs. +1 spotbugs 5m 18s trunk passed       Patch Compile Tests   0 mvndep 0m 27s Maven dependency ordering for patch +1 mvninstall 2m 2s the patch passed +1 compile 12m 18s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 javac 12m 18s the patch passed +1 compile 9m 30s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 +1 javac 9m 30s the patch passed +1 checkstyle 1m 39s the patch passed +1 mvnsite 2m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 shadedclient 16m 21s patch has no errors when building and testing our client artifacts. +1 javadoc 2m 20s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 javadoc 2m 20s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 +1 spotbugs 5m 43s the patch passed       Other Tests   +1 unit 1m 5s hadoop-yarn-api in the patch passed. +1 unit 5m 1s hadoop-yarn-common in the patch passed. +1 unit 1m 12s hadoop-yarn-server-web-proxy in the patch passed. +1 asflicense 0m 41s The patch does not generate ASF License warnings. 138m 46s Subsystem Report/Notes Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/884/artifact/out/Dockerfile JIRA Issue YARN-10720 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13023244/YARN-10720.005.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs xml uname Linux bf7e7f627862 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality personality/hadoop.sh git revision trunk / ff6ec20d845 Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/884/testReport/ Max. process+thread count 567 (vs. ulimit of 5500) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/884/console versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org This message was automatically generated.
            pbacsko Peter Bacsko added a comment -
            	      } catch (InterruptedException e) {
            	        LOG.warn("doGet() interrupted", e);
            	        resp.setStatus(HttpServletResponse.SC_BAD_REQUEST);
            	      }
            	      resp.setStatus(HttpServletResponse.SC_OK);
            	    }
            

            This is not good - you set the response status to SC_BAD_REQUEST only to override it with SC_OK. You need a "return".

            	    try {
            	      servlet.init(config);
            	    } catch (ServletException e) {
            	      LOG.error(e.getMessage());
            	      fail("Failed to init servlet");
            	    }
            
            	    try {
            	      servlet.doGet(request, response);
            	    } catch (ServletException e) {
            	      LOG.error(e.getMessage());
            	      fail("ServletException thrown during doGet.");
            	    }
            	  }
            

            You can remove try-catch here and just add throws ServletException. If that happens for whatever reason, it will be a test error (which is desired - checking if the servlet can init is not the purpose of the test), not a test failure.

            pbacsko Peter Bacsko added a comment - } catch (InterruptedException e) { LOG.warn("doGet() interrupted", e); resp.setStatus(HttpServletResponse.SC_BAD_REQUEST); } resp.setStatus(HttpServletResponse.SC_OK); } This is not good - you set the response status to SC_BAD_REQUEST only to override it with SC_OK . You need a "return". try { servlet.init(config); } catch (ServletException e) { LOG.error(e.getMessage()); fail("Failed to init servlet"); } try { servlet.doGet(request, response); } catch (ServletException e) { LOG.error(e.getMessage()); fail("ServletException thrown during doGet."); } } You can remove try-catch here and just add throws ServletException . If that happens for whatever reason, it will be a test error (which is desired - checking if the servlet can init is not the purpose of the test), not a test failure.
            zhuqi Qi Zhu added a comment -

            Thanks pbacsko for patient review.

            It make sense to me now, updated above in latest patch. 

            zhuqi Qi Zhu added a comment - Thanks pbacsko  for patient review. It make sense to me now, updated above in latest patch. 
            hadoopci Hadoop QA added a comment -
            +1 overall



            Vote Subsystem Runtime Logfile Comment
            0 reexec 1m 23s Docker mode activated.
                  Prechecks  
            +1 dupname 0m 0s No case conflicting files found.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 0m 0s test4tests The patch appears to include 1 new or modified test files.
                  trunk Compile Tests  
            0 mvndep 0m 23s Maven dependency ordering for branch
            +1 mvninstall 23m 44s trunk passed
            +1 compile 10m 11s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 compile 8m 53s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            +1 checkstyle 1m 42s trunk passed
            +1 mvnsite 2m 12s trunk passed
            +1 shadedclient 19m 50s branch has no errors when building and testing our client artifacts.
            +1 javadoc 1m 52s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 javadoc 2m 6s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            0 spotbugs 28m 32s Both FindBugs and SpotBugs are enabled, using SpotBugs.
            +1 spotbugs 4m 48s trunk passed
                  Patch Compile Tests  
            0 mvndep 0m 24s Maven dependency ordering for patch
            +1 mvninstall 1m 37s the patch passed
            +1 compile 10m 12s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 javac 10m 12s the patch passed
            +1 compile 8m 36s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            +1 javac 8m 36s the patch passed
            +1 checkstyle 1m 42s the patch passed
            +1 mvnsite 2m 6s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 shadedclient 15m 19s patch has no errors when building and testing our client artifacts.
            +1 javadoc 1m 51s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
            +1 javadoc 2m 6s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            +1 spotbugs 5m 28s the patch passed
                  Other Tests  
            +1 unit 0m 56s hadoop-yarn-api in the patch passed.
            +1 unit 4m 44s hadoop-yarn-common in the patch passed.
            +1 unit 1m 17s hadoop-yarn-server-web-proxy in the patch passed.
            +1 asflicense 1m 47s The patch does not generate ASF License warnings.
            134m 35s



            Subsystem Report/Notes
            Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/885/artifact/out/Dockerfile
            JIRA Issue YARN-10720
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13023256/YARN-10720.006.patch
            Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs xml
            uname Linux 5d5caf964b63 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality personality/hadoop.sh
            git revision trunk / 6fd0c661b64
            Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
            Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/885/testReport/
            Max. process+thread count 513 (vs. ulimit of 5500)
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn
            Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/885/console
            versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
            Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

            This message was automatically generated.

            hadoopci Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Logfile Comment 0 reexec 1m 23s Docker mode activated.       Prechecks   +1 dupname 0m 0s No case conflicting files found. +1 @author 0m 0s The patch does not contain any @author tags. +1 0m 0s test4tests The patch appears to include 1 new or modified test files.       trunk Compile Tests   0 mvndep 0m 23s Maven dependency ordering for branch +1 mvninstall 23m 44s trunk passed +1 compile 10m 11s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 compile 8m 53s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 +1 checkstyle 1m 42s trunk passed +1 mvnsite 2m 12s trunk passed +1 shadedclient 19m 50s branch has no errors when building and testing our client artifacts. +1 javadoc 1m 52s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 javadoc 2m 6s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 0 spotbugs 28m 32s Both FindBugs and SpotBugs are enabled, using SpotBugs. +1 spotbugs 4m 48s trunk passed       Patch Compile Tests   0 mvndep 0m 24s Maven dependency ordering for patch +1 mvninstall 1m 37s the patch passed +1 compile 10m 12s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 javac 10m 12s the patch passed +1 compile 8m 36s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 +1 javac 8m 36s the patch passed +1 checkstyle 1m 42s the patch passed +1 mvnsite 2m 6s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 shadedclient 15m 19s patch has no errors when building and testing our client artifacts. +1 javadoc 1m 51s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 +1 javadoc 2m 6s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 +1 spotbugs 5m 28s the patch passed       Other Tests   +1 unit 0m 56s hadoop-yarn-api in the patch passed. +1 unit 4m 44s hadoop-yarn-common in the patch passed. +1 unit 1m 17s hadoop-yarn-server-web-proxy in the patch passed. +1 asflicense 1m 47s The patch does not generate ASF License warnings. 134m 35s Subsystem Report/Notes Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/885/artifact/out/Dockerfile JIRA Issue YARN-10720 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/13023256/YARN-10720.006.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs xml uname Linux 5d5caf964b63 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality personality/hadoop.sh git revision trunk / 6fd0c661b64 Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 Test Results https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/885/testReport/ Max. process+thread count 513 (vs. ulimit of 5500) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn Console output https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/885/console versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org This message was automatically generated.
            pbacsko Peter Bacsko added a comment -

            +1

            thanks zhuqi for the patch, committed to trunk.

            pbacsko Peter Bacsko added a comment - +1 thanks zhuqi  for the patch, committed to trunk.
            aajisaka Akira Ajisaka added a comment -

            Cherry-picked to branch-3.3.

            aajisaka Akira Ajisaka added a comment - Cherry-picked to branch-3.3.
            aajisaka Akira Ajisaka added a comment -

            When cherry-picking to branch-3.2, I had to fix some conflicts. Opened https://github.com/apache/hadoop/pull/4102 for testing.

            aajisaka Akira Ajisaka added a comment - When cherry-picking to branch-3.2, I had to fix some conflicts. Opened https://github.com/apache/hadoop/pull/4102 for testing.
            aajisaka Akira Ajisaka added a comment -

            Opened https://github.com/apache/hadoop/pull/4103 for branch-2.10. I'm seeing this issue in a prod cluster, so I want to backport the fix to all the release branches.

            aajisaka Akira Ajisaka added a comment - Opened https://github.com/apache/hadoop/pull/4103 for branch-2.10. I'm seeing this issue in a prod cluster, so I want to backport the fix to all the release branches.
            aajisaka Akira Ajisaka added a comment -

            Backported to branch-3.2 and branch-2.10.

            aajisaka Akira Ajisaka added a comment - Backported to branch-3.2 and branch-2.10.
            stevel@apache.org Steve Loughran added a comment -

            FIxed in 3.3.3; updating fix versions as appropriate

            stevel@apache.org Steve Loughran added a comment - FIxed in 3.3.3; updating fix versions as appropriate

            People

              zhuqi Qi Zhu
              zhuqi Qi Zhu
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h