Details
Description
Following is proxy server show, too many connections from one client, this caused the proxy server hang, and the yarn web can't jump to web proxy.
Following is the AM which is abnormal, but proxy server don't know it is abnormal already, so the connections can't be closed, we should add time out support in proxy server to prevent this. And one abnormal AM may cause hundreds even thousands of connections, it is very heavy.
After i kill the abnormal AM, the proxy server become healthy. This case happened many times in our production clusters, our clusters are huge, and the abnormal AM will be existed in a regular case.
I will add timeout supported in web proxy server in this jira.
cc pbacsko ebadger Jim_Brennan ztang epayne gandras bteke
Attachments
Attachments
- YARN-10720.006.patch
- 11 kB
- Qi Zhu
- YARN-10720.005.patch
- 11 kB
- Qi Zhu
- YARN-10720.004.patch
- 11 kB
- Qi Zhu
- YARN-10720.003.patch
- 12 kB
- Qi Zhu
- YARN-10720.002.patch
- 12 kB
- Qi Zhu
- YARN-10720.001.patch
- 11 kB
- Qi Zhu
- image-2021-03-29-14-05-32-708.png
- 182 kB
- Qi Zhu
- image-2021-03-29-14-04-33-776.png
- 95 kB
- Qi Zhu
Issue Links
- is depended upon by
-
HADOOP-18198 Release Hadoop 3.3.3: hadoop-3.3.2 with some fixes
- Resolved
- links to
Activity
-1 overall |
Vote | Subsystem | Runtime | Logfile | Comment |
---|---|---|---|---|
0 | reexec | 1m 51s | Docker mode activated. | |
Prechecks | ||||
+1 | dupname | 0m 0s | No case conflicting files found. | |
+1 | @author | 0m 0s | The patch does not contain any @author tags. | |
+1 | 0m 0s | test4tests | The patch appears to include 1 new or modified test files. | |
trunk Compile Tests | ||||
0 | mvndep | 2m 4s | Maven dependency ordering for branch | |
+1 | mvninstall | 24m 23s | trunk passed | |
+1 | compile | 10m 41s | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | compile | 9m 8s | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
+1 | checkstyle | 1m 42s | trunk passed | |
+1 | mvnsite | 1m 19s | trunk passed | |
+1 | shadedclient | 18m 21s | branch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 1m 12s | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | javadoc | 1m 5s | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
0 | spotbugs | 23m 20s | Both FindBugs and SpotBugs are enabled, using SpotBugs. | |
+1 | spotbugs | 2m 44s | trunk passed | |
Patch Compile Tests | ||||
0 | mvndep | 0m 21s | Maven dependency ordering for patch | |
+1 | mvninstall | 0m 55s | the patch passed | |
+1 | compile | 9m 10s | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
-1 | javac | 9m 10s | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/862/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt | hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 1 new + 385 unchanged - 0 fixed = 386 total (was 385) |
+1 | compile | 8m 21s | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
-1 | javac | 8m 21s | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/862/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt | hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 1 new + 354 unchanged - 0 fixed = 355 total (was 354) |
+1 | checkstyle | 1m 40s | the patch passed | |
+1 | mvnsite | 1m 14s | the patch passed | |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. | |
+1 | shadedclient | 14m 58s | patch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 1m 7s | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | javadoc | 1m 4s | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
+1 | spotbugs | 3m 9s | the patch passed | |
Other Tests | ||||
+1 | unit | 0m 56s | hadoop-yarn-api in the patch passed. | |
+1 | unit | 1m 14s | hadoop-yarn-server-web-proxy in the patch passed. | |
+1 | asflicense | 0m 41s | The patch does not generate ASF License warnings. | |
119m 25s |
Subsystem | Report/Notes |
---|---|
Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/862/artifact/out/Dockerfile |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13023101/YARN-10720.001.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs |
uname | Linux 4bfe826d84a3 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | personality/hadoop.sh |
git revision | trunk / ea6595d3b68 |
Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
Test Results | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/862/testReport/ |
Max. process+thread count | 516 (vs. ulimit of 5500) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn |
Console output | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/862/console |
versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Logfile | Comment |
---|---|---|---|---|
0 | reexec | 0m 44s | Docker mode activated. | |
Prechecks | ||||
+1 | dupname | 0m 0s | No case conflicting files found. | |
+1 | @author | 0m 0s | The patch does not contain any @author tags. | |
+1 | 0m 0s | test4tests | The patch appears to include 1 new or modified test files. | |
trunk Compile Tests | ||||
0 | mvndep | 1m 48s | Maven dependency ordering for branch | |
+1 | mvninstall | 20m 45s | trunk passed | |
+1 | compile | 9m 30s | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | compile | 8m 36s | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
+1 | checkstyle | 1m 40s | trunk passed | |
+1 | mvnsite | 2m 40s | trunk passed | |
+1 | shadedclient | 17m 56s | branch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 2m 19s | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | javadoc | 2m 30s | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
0 | spotbugs | 27m 35s | Both FindBugs and SpotBugs are enabled, using SpotBugs. | |
+1 | spotbugs | 4m 52s | trunk passed | |
Patch Compile Tests | ||||
0 | mvndep | 0m 26s | Maven dependency ordering for patch | |
+1 | mvninstall | 1m 39s | the patch passed | |
+1 | compile | 8m 30s | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
-1 | javac | 8m 30s | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt | hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 1 new + 384 unchanged - 0 fixed = 385 total (was 384) |
+1 | compile | 7m 49s | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
-1 | javac | 7m 49s | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt | hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 1 new + 355 unchanged - 0 fixed = 356 total (was 355) |
+1 | checkstyle | 1m 39s | the patch passed | |
+1 | mvnsite | 2m 23s | the patch passed | |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. | |
+1 | xml | 0m 2s | The patch has no ill-formed XML file. | |
+1 | shadedclient | 13m 3s | patch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 2m 10s | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | javadoc | 2m 21s | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
+1 | spotbugs | 5m 21s | the patch passed | |
Other Tests | ||||
+1 | unit | 1m 5s | hadoop-yarn-api in the patch passed. | |
+1 | unit | 4m 53s | hadoop-yarn-common in the patch passed. | |
-1 | unit | 1m 18s | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt | hadoop-yarn-server-web-proxy in the patch passed. |
+1 | asflicense | 0m 53s | The patch does not generate ASF License warnings. | |
126m 34s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.webproxy.TestWebAppProxyServlet |
Subsystem | Report/Notes |
---|---|
Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/artifact/out/Dockerfile |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13023102/YARN-10720.002.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs xml |
uname | Linux 1bd152c39c2d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | personality/hadoop.sh |
git revision | trunk / ea6595d3b68 |
Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
Test Results | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/testReport/ |
Max. process+thread count | 716 (vs. ulimit of 5500) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn |
Console output | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/864/console |
versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Logfile | Comment |
---|---|---|---|---|
0 | reexec | 1m 15s | Docker mode activated. | |
Prechecks | ||||
+1 | dupname | 0m 0s | No case conflicting files found. | |
+1 | @author | 0m 0s | The patch does not contain any @author tags. | |
+1 | 0m 0s | test4tests | The patch appears to include 1 new or modified test files. | |
trunk Compile Tests | ||||
0 | mvndep | 0m 20s | Maven dependency ordering for branch | |
+1 | mvninstall | 21m 37s | trunk passed | |
+1 | compile | 9m 41s | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | compile | 11m 8s | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
+1 | checkstyle | 2m 0s | trunk passed | |
+1 | mvnsite | 2m 43s | trunk passed | |
+1 | shadedclient | 22m 12s | branch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 1m 54s | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | javadoc | 2m 6s | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
0 | spotbugs | 30m 52s | Both FindBugs and SpotBugs are enabled, using SpotBugs. | |
+1 | spotbugs | 4m 43s | trunk passed | |
Patch Compile Tests | ||||
0 | mvndep | 0m 20s | Maven dependency ordering for patch | |
+1 | mvninstall | 1m 34s | the patch passed | |
+1 | compile | 9m 15s | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
-1 | javac | 9m 15s | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/866/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt | hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 1 new + 384 unchanged - 0 fixed = 385 total (was 384) |
+1 | compile | 8m 18s | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
-1 | javac | 8m 18s | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/866/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt | hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 1 new + 354 unchanged - 0 fixed = 355 total (was 354) |
+1 | checkstyle | 1m 38s | the patch passed | |
+1 | mvnsite | 2m 6s | the patch passed | |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. | |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. | |
+1 | shadedclient | 15m 8s | patch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 1m 51s | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | javadoc | 2m 2s | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
+1 | spotbugs | 5m 14s | the patch passed | |
Other Tests | ||||
+1 | unit | 0m 56s | hadoop-yarn-api in the patch passed. | |
+1 | unit | 4m 38s | hadoop-yarn-common in the patch passed. | |
+1 | unit | 1m 12s | hadoop-yarn-server-web-proxy in the patch passed. | |
+1 | asflicense | 0m 42s | The patch does not generate ASF License warnings. | |
133m 10s |
Subsystem | Report/Notes |
---|---|
Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/866/artifact/out/Dockerfile |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13023121/YARN-10720.003.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs xml |
uname | Linux fd01596f6811 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | personality/hadoop.sh |
git revision | trunk / ea6595d3b68 |
Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
Test Results | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/866/testReport/ |
Max. process+thread count | 515 (vs. ulimit of 5500) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn |
Console output | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/866/console |
versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
Could you help review this?
It will be a great improvement for the regular proxy server hang problem in our big production clusters, and without this, we should restart the hang web proxy server regularly.
And, i have a question:
If the none() in ExpectedException has been deprecated should be fixed in java doc here.
Thanks.
Thanks zhuqi for the patch.
1. As you said ExpectedException.none() has been deprecated. Either use the new assertThrows() or @Test(expected = SocketTimeoutException.class), I think using the second is easier.
2.
conf.setInt(YarnConfiguration.RM_PROXY_CONNECTION_TIMEOUT, 1 * 1000);
Just write "1000" instead of "1 * 1000".
3.
try { when(response.getOutputStream()).thenReturn(null); } catch (IOException e) { e.printStackTrace(); }
Unnecessary try-catch block. The method already has a throws clause.
4.
@Override protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException { try { Thread.sleep(10 * 1000); } catch (InterruptedException e) { e.printStackTrace(); } resp.setStatus(HttpServletResponse.SC_OK); }
Maybe a minor thing, but if you catch InterruptedException, don't just print the stack trace, log it with LOG.warn("doGet() interrupted", e). In this case, I'd also return with HttpServletResponse.SC_BAD_REQUEST.
5.
<description>The web proxy connection timeout, default is 60s(60 * 1000ms).</description>
This already goes to yarn-default.xml, so you can omit the part "default is 60s(60 * 1000ms)" and just write "The web proxy connection timeout".
Thanks a lot pbacsko for valid suggestions.
I have updated in latest patch.
+1 overall |
Vote | Subsystem | Runtime | Logfile | Comment |
---|---|---|---|---|
0 | reexec | 0m 53s | Docker mode activated. | |
Prechecks | ||||
+1 | dupname | 0m 0s | No case conflicting files found. | |
+1 | @author | 0m 0s | The patch does not contain any @author tags. | |
+1 | 0m 0s | test4tests | The patch appears to include 1 new or modified test files. | |
trunk Compile Tests | ||||
0 | mvndep | 1m 52s | Maven dependency ordering for branch | |
+1 | mvninstall | 19m 47s | trunk passed | |
+1 | compile | 8m 59s | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | compile | 7m 49s | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
+1 | checkstyle | 1m 44s | trunk passed | |
+1 | mvnsite | 2m 36s | trunk passed | |
+1 | shadedclient | 17m 42s | branch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 2m 18s | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | javadoc | 2m 32s | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
0 | spotbugs | 27m 21s | Both FindBugs and SpotBugs are enabled, using SpotBugs. | |
+1 | spotbugs | 4m 52s | trunk passed | |
Patch Compile Tests | ||||
0 | mvndep | 0m 26s | Maven dependency ordering for patch | |
+1 | mvninstall | 1m 38s | the patch passed | |
+1 | compile | 8m 25s | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | javac | 8m 25s | the patch passed | |
+1 | compile | 7m 48s | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
+1 | javac | 7m 48s | the patch passed | |
-0 | checkstyle | 1m 41s | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/883/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt | hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 258 unchanged - 0 fixed = 260 total (was 258) |
+1 | mvnsite | 2m 23s | the patch passed | |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. | |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. | |
+1 | shadedclient | 13m 3s | patch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 2m 13s | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | javadoc | 2m 24s | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
+1 | spotbugs | 5m 21s | the patch passed | |
Other Tests | ||||
+1 | unit | 1m 4s | hadoop-yarn-api in the patch passed. | |
+1 | unit | 4m 54s | hadoop-yarn-common in the patch passed. | |
+1 | unit | 1m 20s | hadoop-yarn-server-web-proxy in the patch passed. | |
+1 | asflicense | 0m 49s | The patch does not generate ASF License warnings. | |
124m 15s |
Subsystem | Report/Notes |
---|---|
Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/883/artifact/out/Dockerfile |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13023234/YARN-10720.004.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs xml |
uname | Linux 03bbeb7415a2 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | personality/hadoop.sh |
git revision | trunk / ff6ec20d845 |
Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
Test Results | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/883/testReport/ |
Max. process+thread count | 597 (vs. ulimit of 5500) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn |
Console output | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/883/console |
versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
+1 overall |
Vote | Subsystem | Runtime | Logfile | Comment |
---|---|---|---|---|
0 | reexec | 1m 29s | Docker mode activated. | |
Prechecks | ||||
+1 | dupname | 0m 0s | No case conflicting files found. | |
+1 | @author | 0m 0s | The patch does not contain any @author tags. | |
+1 | 0m 0s | test4tests | The patch appears to include 1 new or modified test files. | |
trunk Compile Tests | ||||
0 | mvndep | 1m 36s | Maven dependency ordering for branch | |
+1 | mvninstall | 22m 15s | trunk passed | |
+1 | compile | 9m 46s | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | compile | 8m 14s | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
+1 | checkstyle | 1m 42s | trunk passed | |
+1 | mvnsite | 2m 14s | trunk passed | |
+1 | shadedclient | 19m 32s | branch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 1m 55s | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | javadoc | 2m 8s | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
0 | spotbugs | 28m 47s | Both FindBugs and SpotBugs are enabled, using SpotBugs. | |
+1 | spotbugs | 5m 18s | trunk passed | |
Patch Compile Tests | ||||
0 | mvndep | 0m 27s | Maven dependency ordering for patch | |
+1 | mvninstall | 2m 2s | the patch passed | |
+1 | compile | 12m 18s | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | javac | 12m 18s | the patch passed | |
+1 | compile | 9m 30s | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
+1 | javac | 9m 30s | the patch passed | |
+1 | checkstyle | 1m 39s | the patch passed | |
+1 | mvnsite | 2m 9s | the patch passed | |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. | |
+1 | xml | 0m 2s | The patch has no ill-formed XML file. | |
+1 | shadedclient | 16m 21s | patch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 2m 20s | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | javadoc | 2m 20s | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
+1 | spotbugs | 5m 43s | the patch passed | |
Other Tests | ||||
+1 | unit | 1m 5s | hadoop-yarn-api in the patch passed. | |
+1 | unit | 5m 1s | hadoop-yarn-common in the patch passed. | |
+1 | unit | 1m 12s | hadoop-yarn-server-web-proxy in the patch passed. | |
+1 | asflicense | 0m 41s | The patch does not generate ASF License warnings. | |
138m 46s |
Subsystem | Report/Notes |
---|---|
Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/884/artifact/out/Dockerfile |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13023244/YARN-10720.005.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs xml |
uname | Linux bf7e7f627862 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | personality/hadoop.sh |
git revision | trunk / ff6ec20d845 |
Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
Test Results | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/884/testReport/ |
Max. process+thread count | 567 (vs. ulimit of 5500) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn |
Console output | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/884/console |
versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
} catch (InterruptedException e) { LOG.warn("doGet() interrupted", e); resp.setStatus(HttpServletResponse.SC_BAD_REQUEST); } resp.setStatus(HttpServletResponse.SC_OK); }
This is not good - you set the response status to SC_BAD_REQUEST only to override it with SC_OK. You need a "return".
try { servlet.init(config); } catch (ServletException e) { LOG.error(e.getMessage()); fail("Failed to init servlet"); } try { servlet.doGet(request, response); } catch (ServletException e) { LOG.error(e.getMessage()); fail("ServletException thrown during doGet."); } }
You can remove try-catch here and just add throws ServletException. If that happens for whatever reason, it will be a test error (which is desired - checking if the servlet can init is not the purpose of the test), not a test failure.
Thanks pbacsko for patient review.
It make sense to me now, updated above in latest patch.
+1 overall |
Vote | Subsystem | Runtime | Logfile | Comment |
---|---|---|---|---|
0 | reexec | 1m 23s | Docker mode activated. | |
Prechecks | ||||
+1 | dupname | 0m 0s | No case conflicting files found. | |
+1 | @author | 0m 0s | The patch does not contain any @author tags. | |
+1 | 0m 0s | test4tests | The patch appears to include 1 new or modified test files. | |
trunk Compile Tests | ||||
0 | mvndep | 0m 23s | Maven dependency ordering for branch | |
+1 | mvninstall | 23m 44s | trunk passed | |
+1 | compile | 10m 11s | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | compile | 8m 53s | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
+1 | checkstyle | 1m 42s | trunk passed | |
+1 | mvnsite | 2m 12s | trunk passed | |
+1 | shadedclient | 19m 50s | branch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 1m 52s | trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | javadoc | 2m 6s | trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
0 | spotbugs | 28m 32s | Both FindBugs and SpotBugs are enabled, using SpotBugs. | |
+1 | spotbugs | 4m 48s | trunk passed | |
Patch Compile Tests | ||||
0 | mvndep | 0m 24s | Maven dependency ordering for patch | |
+1 | mvninstall | 1m 37s | the patch passed | |
+1 | compile | 10m 12s | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | javac | 10m 12s | the patch passed | |
+1 | compile | 8m 36s | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
+1 | javac | 8m 36s | the patch passed | |
+1 | checkstyle | 1m 42s | the patch passed | |
+1 | mvnsite | 2m 6s | the patch passed | |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. | |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. | |
+1 | shadedclient | 15m 19s | patch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 1m 51s | the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 | |
+1 | javadoc | 2m 6s | the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 | |
+1 | spotbugs | 5m 28s | the patch passed | |
Other Tests | ||||
+1 | unit | 0m 56s | hadoop-yarn-api in the patch passed. | |
+1 | unit | 4m 44s | hadoop-yarn-common in the patch passed. | |
+1 | unit | 1m 17s | hadoop-yarn-server-web-proxy in the patch passed. | |
+1 | asflicense | 1m 47s | The patch does not generate ASF License warnings. | |
134m 35s |
Subsystem | Report/Notes |
---|---|
Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/885/artifact/out/Dockerfile |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13023256/YARN-10720.006.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle spotbugs xml |
uname | Linux 5d5caf964b63 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | personality/hadoop.sh |
git revision | trunk / 6fd0c661b64 |
Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
Test Results | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/885/testReport/ |
Max. process+thread count | 513 (vs. ulimit of 5500) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: hadoop-yarn-project/hadoop-yarn |
Console output | https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/885/console |
versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
When cherry-picking to branch-3.2, I had to fix some conflicts. Opened https://github.com/apache/hadoop/pull/4102 for testing.
Opened https://github.com/apache/hadoop/pull/4103 for branch-2.10. I'm seeing this issue in a prod cluster, so I want to backport the fix to all the release branches.
cc pbacsko ebadger Jim_Brennan ztang epayne gandras bteke brahmareddy
Could you help review this?
Tested in our test cluster, works well.
Thanks.