Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
Reviewed
Description
Currently, TFile log reader is used to read aggregated log in YARN. We need to add an abstract layer, and pick up the correct log reader based on the configuration.
Attachments
Attachments
- YARN-6877-branch-2.001.patch
- 82 kB
- Xuan Gong
- YARN-6877-branch-2.002.patch
- 115 kB
- Xuan Gong
- YARN-6877-trunk.001.patch
- 84 kB
- Xuan Gong
- YARN-6877-trunk.002.patch
- 89 kB
- Xuan Gong
- YARN-6877-trunk.003.patch
- 115 kB
- Xuan Gong
- YARN-6877-trunk.004.patch
- 115 kB
- Xuan Gong
- YARN-6877-trunk.005.patch
- 119 kB
- Xuan Gong
- YARN-6877-trunk.006.patch
- 116 kB
- Xuan Gong
- YARN-6877-trunk.007.patch
- 116 kB
- Xuan Gong
- YARN-6877-trunk.008.patch
- 116 kB
- Xuan Gong
Issue Links
Activity
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 18s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 17s | Maven dependency ordering for branch |
+1 | mvninstall | 16m 43s | trunk passed |
+1 | compile | 11m 10s | trunk passed |
+1 | checkstyle | 1m 12s | trunk passed |
+1 | mvnsite | 2m 20s | trunk passed |
-1 | findbugs | 0m 56s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 1m 57s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 12s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 51s | the patch passed |
+1 | compile | 6m 36s | the patch passed |
+1 | javac | 6m 36s | the patch passed |
-0 | checkstyle | 1m 7s | hadoop-yarn-project/hadoop-yarn: The patch generated 11 new + 318 unchanged - 11 fixed = 329 total (was 329) |
+1 | mvnsite | 2m 13s | the patch passed |
-1 | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
-1 | findbugs | 1m 31s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
-1 | javadoc | 0m 47s | hadoop-yarn-common in the patch failed. |
Other Tests | |||
+1 | unit | 2m 30s | hadoop-yarn-common in the patch passed. |
+1 | unit | 13m 31s | hadoop-yarn-server-nodemanager in the patch passed. |
+1 | unit | 3m 6s | hadoop-yarn-server-applicationhistoryservice in the patch passed. |
-1 | unit | 4m 56s | hadoop-yarn-client in the patch failed. |
+1 | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
89m 1s |
Reason | Tests |
---|---|
FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
org.apache.hadoop.yarn.logaggregation.filecontroller.tfile.TFileAggregatedLogsBlock.render(HtmlBlock$Block) might ignore java.lang.Exception At TFileAggregatedLogsBlock.java:At TFileAggregatedLogsBlock.java:[line 113] | |
Failed junit tests | hadoop.yarn.client.TestRMFailover |
hadoop.yarn.client.api.impl.TestAMRMProxy | |
hadoop.yarn.client.cli.TestLogsCLI | |
hadoop.yarn.client.TestApplicationClientProtocolOnHA | |
hadoop.yarn.client.api.impl.TestNMClient | |
hadoop.yarn.client.cli.TestYarnCLI | |
hadoop.yarn.client.api.impl.TestYarnClient | |
hadoop.yarn.client.api.impl.TestAMRMClient | |
hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation | |
hadoop.yarn.client.api.impl.TestDistributedScheduling |
This message was automatically generated.
Fix the findbugs warning and checkstyle issue.
I can pass all the testcases on my local machine. Kick the Jenkins again to verify.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 16s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 42s | Maven dependency ordering for branch |
+1 | mvninstall | 14m 11s | trunk passed |
+1 | compile | 8m 58s | trunk passed |
+1 | checkstyle | 1m 0s | trunk passed |
+1 | mvnsite | 2m 12s | trunk passed |
-1 | findbugs | 0m 50s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 1m 49s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 11s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 36s | the patch passed |
+1 | compile | 5m 36s | the patch passed |
+1 | javac | 5m 36s | the patch passed |
-0 | checkstyle | 0m 59s | hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 315 unchanged - 11 fixed = 321 total (was 326) |
+1 | mvnsite | 2m 7s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
+1 | findbugs | 3m 44s | the patch passed |
-1 | javadoc | 0m 42s | hadoop-yarn-common in the patch failed. |
Other Tests | |||
+1 | unit | 2m 29s | hadoop-yarn-common in the patch passed. |
+1 | unit | 13m 32s | hadoop-yarn-server-nodemanager in the patch passed. |
+1 | unit | 3m 6s | hadoop-yarn-server-applicationhistoryservice in the patch passed. |
-1 | unit | 4m 58s | hadoop-yarn-client in the patch failed. |
+1 | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
81m 39s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMProxy |
hadoop.yarn.client.api.impl.TestDistributedScheduling | |
hadoop.yarn.client.cli.TestLogsCLI | |
hadoop.yarn.client.TestApplicationClientProtocolOnHA | |
hadoop.yarn.client.api.impl.TestAMRMClient | |
hadoop.yarn.client.api.impl.TestNMClient | |
hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation | |
hadoop.yarn.client.TestRMFailover | |
hadoop.yarn.client.cli.TestYarnCLI | |
hadoop.yarn.client.api.impl.TestYarnClient |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:14b5c93 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12883986/YARN-6877-trunk.004.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml |
uname | Linux 814466f82dd8 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / ae7abad |
Default Java | 1.8.0_144 |
findbugs | v3.1.0-RC1 |
findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/17146/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/17146/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/17146/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/17146/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17146/testReport/ |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/17146/console |
Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 18s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 46s | Maven dependency ordering for branch |
+1 | mvninstall | 16m 13s | trunk passed |
+1 | compile | 10m 50s | trunk passed |
+1 | checkstyle | 1m 6s | trunk passed |
+1 | mvnsite | 2m 25s | trunk passed |
-1 | findbugs | 0m 56s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 1m 59s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 10s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 52s | the patch passed |
+1 | compile | 6m 46s | the patch passed |
+1 | javac | 6m 46s | the patch passed |
-0 | checkstyle | 1m 1s | hadoop-yarn-project/hadoop-yarn: The patch generated 9 new + 315 unchanged - 11 fixed = 324 total (was 326) |
+1 | mvnsite | 2m 14s | the patch passed |
-1 | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
+1 | xml | 0m 2s | The patch has no ill-formed XML file. |
+1 | findbugs | 4m 18s | the patch passed |
-1 | javadoc | 0m 43s | hadoop-yarn-common in the patch failed. |
Other Tests | |||
+1 | unit | 2m 41s | hadoop-yarn-common in the patch passed. |
+1 | unit | 14m 9s | hadoop-yarn-server-nodemanager in the patch passed. |
+1 | unit | 3m 24s | hadoop-yarn-server-applicationhistoryservice in the patch passed. |
-1 | unit | 5m 42s | hadoop-yarn-client in the patch failed. |
+1 | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
90m 36s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMProxy |
hadoop.yarn.client.api.impl.TestDistributedScheduling | |
hadoop.yarn.client.cli.TestLogsCLI | |
hadoop.yarn.client.TestApplicationClientProtocolOnHA | |
hadoop.yarn.client.api.impl.TestAMRMClient | |
hadoop.yarn.client.api.impl.TestNMClient | |
hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation | |
hadoop.yarn.client.TestRMFailover | |
hadoop.yarn.client.cli.TestYarnCLI | |
hadoop.yarn.client.api.impl.TestYarnClient |
This message was automatically generated.
Thanks xgong for working on a patch.
The patch is huge. However, mostly are refactoring format binding methods from LogCLIHelpers, LogToolUtils and AggregatedLogsBlock to new created extensible classes, so still straightforward. Some review comments:
It seems we have two AggregatedLogsBlock now, both are extended from HtmlBlock. One is under filecontroller package (new) while the other in web app package (old):
+@InterfaceAudience.LimitedPrivate({"YARN", "MapReduce"}) +public abstract class AggregatedLogsBlock extends HtmlBlock {
What's use case for both AggregatedLogsBlock here? Does old one get dropped/deprecated? If not, we may need to think some other name for the new class to get rid of confusion here.
Some minor comments:
In TestLogsCLI.java,
- Configuration configuration = new Configuration(); + Configuration configuration = new YarnConfiguration();
Why we need this change? I think both should be working here. Do I miss anything?
In many places, the code are missing space between between '}' and catch
}catch (NumberFormatException ne) {
In LogCLIHelpers.java, I saw we are mixing using err.println() and System.err.println(). Both way should works fine but better to keep consistent here.
seems like no change in AggregatedLogsBlockForTest.java. If so, better remove it.
In LogCLIHelpers.java, I saw we are mixing using err.println() and System.err.println(). Both way should works fine but better to keep consistent here.
This is because some of the function use (PrintStream err) as parameter input, so err.println() will be used. For the functions which do not have (PrintStream err) as parameter, we use System.err.println()
Uploaded a new patch for other comments.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 23s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 48s | Maven dependency ordering for branch |
+1 | mvninstall | 13m 37s | trunk passed |
+1 | compile | 8m 53s | trunk passed |
+1 | checkstyle | 1m 1s | trunk passed |
+1 | mvnsite | 2m 16s | trunk passed |
-1 | findbugs | 0m 50s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 1m 59s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 11s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 31s | the patch passed |
+1 | compile | 5m 36s | the patch passed |
+1 | javac | 5m 36s | the patch passed |
-0 | checkstyle | 0m 58s | hadoop-yarn-project/hadoop-yarn: The patch generated 8 new + 314 unchanged - 11 fixed = 322 total (was 325) |
+1 | mvnsite | 2m 11s | the patch passed |
-1 | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
+1 | findbugs | 3m 45s | the patch passed |
-1 | javadoc | 0m 42s | hadoop-yarn-common in the patch failed. |
Other Tests | |||
+1 | unit | 2m 40s | hadoop-yarn-common in the patch passed. |
+1 | unit | 13m 58s | hadoop-yarn-server-nodemanager in the patch passed. |
+1 | unit | 3m 7s | hadoop-yarn-server-applicationhistoryservice in the patch passed. |
-1 | unit | 4m 43s | hadoop-yarn-client in the patch failed. |
+1 | asflicense | 0m 31s | The patch does not generate ASF License warnings. |
82m 31s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMClient |
hadoop.yarn.client.api.impl.TestDistributedScheduling | |
hadoop.yarn.client.api.impl.TestYarnClient | |
hadoop.yarn.client.api.impl.TestAMRMProxy | |
hadoop.yarn.client.TestApplicationClientProtocolOnHA | |
hadoop.yarn.client.cli.TestLogsCLI | |
hadoop.yarn.client.TestRMFailover | |
hadoop.yarn.client.cli.TestYarnCLI | |
hadoop.yarn.client.api.impl.TestNMClient | |
hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation |
This message was automatically generated.
Thanks Xuan for uploading a patch!
Looks like in latest patch, issue still exists for my previous comments:
public void testFetchApplictionLogsHar() throws Exception { String remoteLogRootDir = "target/logs/"; - Configuration configuration = new Configuration(); + Configuration configuration = new YarnConfiguration();
Still lacking space between } and catch in some places:
}catch (NumberFormatException ne) {
Some new comments:
In LogAggregationWebUtils.java, verifyAndGetLogStartIndex() is better to change to getLogStartIndex(). Normally we should check valid of input if we start with verify*, like original methods - verifyAndGetLogLimits(), but here we just simply throw exceptions (if input is illegal) and rely on caller to catch exception and handle. I think call getLogStartIndex() should be better. Same comments for verifyAndGetLogEndIndex(). But other verfiy* methods should be fine - at least no exception for caller to catch and handle.
In LogCLIHelpers.dumpAContainerLogsForLogType():
+ return 0; + } catch (IOException ex) { + System.err.println(ex.getMessage()); return -1; }
It looks like we swallow IOException here instead of throw it out as previous behavior. Does this expected? If not, better to keep the original way.
The same comments for dumpAllContainersLogs().
I miss one item in previous review for LogAggregationTFileController.java,
+ @Override + public void closeWriter() { + this.writer.close(); + }
Do we need to set writer here to null after we close it which is consistent with previous behavior? The previous intention seems like to keep minimum retain memory. The reason is still valid here. Isn't it?
Other looks fine to me.
btw, Findbugs warning shouldn't be related. Can you please check checkstyle, javadoc warnings here?
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 29s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 20s | Maven dependency ordering for branch |
+1 | mvninstall | 17m 52s | trunk passed |
+1 | compile | 9m 41s | trunk passed |
+1 | checkstyle | 1m 1s | trunk passed |
+1 | mvnsite | 2m 12s | trunk passed |
-1 | findbugs | 0m 52s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 1m 52s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 11s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 40s | the patch passed |
+1 | compile | 6m 2s | the patch passed |
+1 | javac | 6m 2s | the patch passed |
-0 | checkstyle | 1m 0s | hadoop-yarn-project/hadoop-yarn: The patch generated 8 new + 314 unchanged - 11 fixed = 322 total (was 325) |
+1 | mvnsite | 2m 14s | the patch passed |
-1 | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
+1 | findbugs | 4m 1s | the patch passed |
-1 | javadoc | 0m 44s | hadoop-yarn-common in the patch failed. |
Other Tests | |||
+1 | unit | 2m 34s | hadoop-yarn-common in the patch passed. |
+1 | unit | 13m 45s | hadoop-yarn-server-nodemanager in the patch passed. |
+1 | unit | 3m 8s | hadoop-yarn-server-applicationhistoryservice in the patch passed. |
-1 | unit | 5m 14s | hadoop-yarn-client in the patch failed. |
+1 | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
87m 35s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.client.cli.TestLogsCLI |
hadoop.yarn.client.TestRMFailover | |
hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation | |
hadoop.yarn.client.api.impl.TestDistributedScheduling | |
hadoop.yarn.client.TestApplicationMasterServiceProtocolForTimelineV2 | |
hadoop.yarn.client.TestApplicationClientProtocolOnHA | |
hadoop.yarn.client.api.impl.TestAMRMClient | |
hadoop.yarn.client.api.impl.TestNMClient | |
hadoop.yarn.client.cli.TestYarnCLI | |
hadoop.yarn.client.api.impl.TestAMRMProxy | |
hadoop.yarn.client.api.impl.TestYarnClient |
This message was automatically generated.
Thanks xgong for updating the patch. 007 patch looks good in overall. Only one small NIT here:
In LogCLIHelpers.dumpAContainerLogsForLogTypeWithoutNodeId() we still catch IOException instead of throw it out.
Will +1 after fixing this.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 21s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 10s | Maven dependency ordering for branch |
+1 | mvninstall | 15m 2s | trunk passed |
+1 | compile | 10m 19s | trunk passed |
+1 | checkstyle | 1m 2s | trunk passed |
+1 | mvnsite | 2m 11s | trunk passed |
-1 | findbugs | 0m 59s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 1m 56s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 10s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 37s | the patch passed |
+1 | compile | 6m 27s | the patch passed |
+1 | javac | 6m 27s | the patch passed |
-0 | checkstyle | 1m 5s | hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 315 unchanged - 11 fixed = 317 total (was 326) |
+1 | mvnsite | 2m 8s | the patch passed |
+1 | whitespace | 0m 1s | The patch has no whitespace issues. |
+1 | xml | 0m 2s | The patch has no ill-formed XML file. |
+1 | findbugs | 3m 54s | the patch passed |
-1 | javadoc | 0m 47s | hadoop-yarn-common in the patch failed. |
Other Tests | |||
+1 | unit | 2m 44s | hadoop-yarn-common in the patch passed. |
+1 | unit | 14m 9s | hadoop-yarn-server-nodemanager in the patch passed. |
+1 | unit | 3m 10s | hadoop-yarn-server-applicationhistoryservice in the patch passed. |
-1 | unit | 5m 14s | hadoop-yarn-client in the patch failed. |
+1 | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
86m 23s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.client.TestApplicationClientProtocolOnHA |
hadoop.yarn.client.TestApplicationMasterServiceProtocolForTimelineV2 | |
hadoop.yarn.client.cli.TestYarnCLI | |
hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation | |
hadoop.yarn.client.TestRMFailover | |
hadoop.yarn.client.api.impl.TestAMRMClient | |
hadoop.yarn.client.cli.TestLogsCLI | |
hadoop.yarn.client.api.impl.TestDistributedScheduling | |
hadoop.yarn.client.api.impl.TestNMClient | |
hadoop.yarn.client.api.impl.TestAMRMProxy | |
hadoop.yarn.client.api.impl.TestYarnClient |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:71bbb86 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12884698/YARN-6877-trunk.007.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml |
uname | Linux 2397dd07a0a1 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / d4417da |
Default Java | 1.8.0_144 |
findbugs | v3.1.0-RC1 |
findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/17237/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/17237/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/17237/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/17237/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17237/testReport/ |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/17237/console |
Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 22m 18s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 0m 48s | Maven dependency ordering for branch |
+1 | mvninstall | 15m 22s | trunk passed |
+1 | compile | 11m 55s | trunk passed |
+1 | checkstyle | 1m 9s | trunk passed |
+1 | mvnsite | 2m 27s | trunk passed |
-1 | findbugs | 0m 58s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. |
+1 | javadoc | 1m 59s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 11s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 56s | the patch passed |
+1 | compile | 6m 57s | the patch passed |
+1 | javac | 6m 57s | the patch passed |
-0 | checkstyle | 1m 4s | hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 314 unchanged - 11 fixed = 316 total (was 325) |
+1 | mvnsite | 2m 30s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 2s | The patch has no ill-formed XML file. |
+1 | findbugs | 4m 16s | the patch passed |
-1 | javadoc | 0m 47s | hadoop-yarn-common in the patch failed. |
Other Tests | |||
+1 | unit | 2m 35s | hadoop-yarn-common in the patch passed. |
+1 | unit | 14m 54s | hadoop-yarn-server-nodemanager in the patch passed. |
+1 | unit | 3m 56s | hadoop-yarn-server-applicationhistoryservice in the patch passed. |
-1 | unit | 5m 58s | hadoop-yarn-client in the patch failed. |
+1 | asflicense | 0m 33s | The patch does not generate ASF License warnings. |
115m 32s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.client.cli.TestYarnCLI |
hadoop.yarn.client.api.impl.TestNMClient | |
hadoop.yarn.client.api.impl.TestAMRMProxy | |
hadoop.yarn.client.api.impl.TestAMRMClient | |
hadoop.yarn.client.TestApplicationMasterServiceProtocolForTimelineV2 | |
hadoop.yarn.client.cli.TestLogsCLI | |
hadoop.yarn.client.api.impl.TestYarnClient | |
hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation | |
hadoop.yarn.client.api.impl.TestDistributedScheduling | |
hadoop.yarn.client.TestApplicationClientProtocolOnHA | |
hadoop.yarn.client.TestRMFailover |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:71bbb86 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12884791/YARN-6877-trunk.008.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml |
uname | Linux 7e41e0cdcc85 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / d4417da |
Default Java | 1.8.0_144 |
findbugs | v3.1.0-RC1 |
findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/17243/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/17243/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/17243/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/17243/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17243/testReport/ |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/17243/console |
Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
The test failures (and findbugs warning) shouldn't be related to this patch. Javadoc report is not clear on which new warnings get involved so suspect it is just meaningless warning. +1. Committing it in.
I have commit the patch to trunk. However, trying to backport to branch-2 end up with several file conflicts. xgong, can you upload a patch against branch-2? Thx!
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12287 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12287/)
YARN-6877. Create an abstract log reader for extendability. Contributed (junping_du: rev 91cc070d67533ebb3325b982eba2135e0d175a82)
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestAppLogAggregatorImpl.java
- (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/tfile/LogAggregationTFileController.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogsBlock.java
- (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationHtmlBlock.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileController.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebServices.java
- (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/tfile/TFileAggregatedLogsBlock.java
- (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/tfile/package-info.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
- (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogAggregationWebUtils.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/filecontroller/TestLogAggregationFileControllerFactory.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebServices.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/log/AggregatedLogsBlock.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogToolUtils.java
- (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationTFileController.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 25s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
branch-2 Compile Tests | |||
0 | mvndep | 0m 25s | Maven dependency ordering for branch |
+1 | mvninstall | 7m 30s | branch-2 passed |
+1 | compile | 2m 14s | branch-2 passed with JDK v1.8.0_144 |
+1 | compile | 2m 27s | branch-2 passed with JDK v1.7.0_151 |
+1 | checkstyle | 0m 45s | branch-2 passed |
+1 | mvnsite | 2m 2s | branch-2 passed |
+1 | findbugs | 3m 30s | branch-2 passed |
+1 | javadoc | 1m 22s | branch-2 passed with JDK v1.8.0_144 |
+1 | javadoc | 1m 28s | branch-2 passed with JDK v1.7.0_151 |
Patch Compile Tests | |||
0 | mvndep | 0m 11s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 43s | the patch passed |
+1 | compile | 2m 4s | the patch passed with JDK v1.8.0_144 |
+1 | javac | 2m 4s | the patch passed |
+1 | compile | 2m 31s | the patch passed with JDK v1.7.0_151 |
+1 | javac | 2m 31s | the patch passed |
-0 | checkstyle | 0m 46s | hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 309 unchanged - 15 fixed = 311 total (was 324) |
+1 | mvnsite | 1m 57s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
+1 | findbugs | 4m 10s | the patch passed |
-1 | javadoc | 0m 25s | hadoop-yarn-common in the patch failed with JDK v1.8.0_144. |
-1 | javadoc | 0m 31s | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.7.0_151 with JDK v1.7.0_151 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
Other Tests | |||
+1 | unit | 2m 24s | hadoop-yarn-common in the patch passed with JDK v1.7.0_151. |
-1 | unit | 13m 6s | hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_151. |
+1 | unit | 3m 18s | hadoop-yarn-server-applicationhistoryservice in the patch passed with JDK v1.7.0_151. |
-1 | unit | 63m 15s | hadoop-yarn-client in the patch failed with JDK v1.7.0_151. |
+1 | asflicense | 0m 25s | The patch does not generate ASF License warnings. |
151m 2s |
Reason | Tests |
---|---|
JDK v1.8.0_144 Failed junit tests | hadoop.yarn.server.nodemanager.webapp.TestNMWebServer |
hadoop.yarn.client.TestApplicationClientProtocolOnHA | |
hadoop.yarn.client.cli.TestYarnCLI | |
hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation | |
hadoop.yarn.client.TestRMFailover | |
hadoop.yarn.client.api.impl.TestAMRMClient | |
hadoop.yarn.client.cli.TestLogsCLI | |
hadoop.yarn.client.api.impl.TestDistributedScheduling | |
hadoop.yarn.client.api.impl.TestNMClient | |
hadoop.yarn.client.api.impl.TestAMRMProxy | |
hadoop.yarn.client.api.impl.TestYarnClient | |
JDK v1.7.0_151 Failed junit tests | hadoop.yarn.server.nodemanager.webapp.TestNMWebServer |
hadoop.yarn.client.cli.TestYarnCLI | |
hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation | |
hadoop.yarn.client.cli.TestLogsCLI | |
hadoop.yarn.client.api.impl.TestDistributedScheduling | |
hadoop.yarn.client.api.impl.TestNMClient | |
hadoop.yarn.client.api.impl.TestAMRMProxy | |
hadoop.yarn.client.api.impl.TestYarnClient | |
JDK v1.7.0_151 Timed out junit tests | org.apache.hadoop.yarn.client.TestRMFailover |
org.apache.hadoop.yarn.client.TestApplicationMasterServiceProtocolOnHA | |
org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA | |
org.apache.hadoop.yarn.client.api.impl.TestAMRMClient |
This message was automatically generated.
Thanks xgong. The test failures should be unrelated. commit the patch to branch-2 as well.
Allen said in dev MLs:
Looks like
YARN-6877is crashing JDK8 javadoc. Maven stops processing and errors out before even giving a build error/success. Reverting the patch makes things work again. Anyway, Yetus caught it, warned about it continuously, but it was still committed.
In short, the following warning must not be ignored:
-1 javadoc 0m 25s hadoop-yarn-common in the patch failed with JDK v1.8.0_144.
Hi junping_du and xgong, would you fix it?
Thanks ajisakaa for noticing us.
In short, the following warning must not be ignored.
As I mentioned above, I noticed this issue - "Javadoc report is not clear on which new warnings get involved so suspect it is just meaningless warning." But look at the javadoc warnings link, you will find thousands of warnings/errors there which confuse me that it is just meaning less info.
Hi Junping Du and Xuan Gong, would you fix it?
Of course. xgong, can you take a quick look?
Thank you for the quick reply!
The branch-2 patch or the trunk patch breaks the build?
The both.
As I mentioned above, I noticed this issue - "Javadoc report is not clear on which new warnings get involved so suspect it is just meaningless warning." But look at the javadoc warnings link, you will find thousands of warnings/errors there which confuse me that it is just meaning less info.
Agreed that it is confusing, however, in the report, the command failed by error. For example:
- hadoop-yarn-common module has some errors, so it failed: https://builds.apache.org/job/PreCommit-YARN-Build/17243/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
- hadoop-yarn-client has many warnings but no error, so it succeeded: https://builds.apache.org/job/PreCommit-YARN-Build/17243/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
If there is an error, it fails. Therefore you need to fix all the errors.
Ideally, all the remaining warnings should be fixed as well to avoid this type of issue.
It is very tricky to find out real errors among thousands of ERROR messages, but I think finally I found two that is really error:
... [ERROR] /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileController.java:231: error: @param name not found [ERROR] * @param the aggregatedLog path. ... [ERROR] /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileController.java:242: error: @param name not found [ERROR] * @param the aggregatedLog path.
xgong, can you double check any other java doc errors?
Those log aggregation related errors will be fixed with https://issues.apache.org/jira/browse/YARN-7148 which I have already uploaded a patch.
YARN-6877This message was automatically generated.