Jian He added a comment - 19/May/16 00:08 move the check and init part to the common LogAggregationService init.

Hadoop QA added a comment - 19/May/16 01:23 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 45s trunk passed +1 compile 0m 25s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 39s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 27s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed -1 checkstyle 0m 17s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 6 new + 194 unchanged - 5 fixed = 200 total (was 199) +1 mvnsite 0m 27s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 51s the patch passed +1 javadoc 0m 19s the patch passed -1 unit 11m 42s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 24m 48s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804834/YARN-5112.patch JIRA Issue YARN-5112 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d9c2eba187fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 010e6ac Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11536/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11536/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11536/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11536/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11536/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

Hadoop QA added a comment - 19/May/16 06:43 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 35s trunk passed +1 compile 0m 28s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 10s trunk passed +1 findbugs 0m 43s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 23s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed -1 checkstyle 0m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 6 new + 193 unchanged - 5 fixed = 199 total (was 198) +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 53s the patch passed +1 javadoc 0m 16s the patch passed -1 unit 11m 28s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 15s Patch does not generate ASF License warnings. 26m 16s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804834/YARN-5112.patch JIRA Issue YARN-5112 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f3ecde06b25a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1597630 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11547/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11547/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11547/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11547/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11547/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

Junping Du added a comment - 19/May/16 12:38 Thanks Jian He for reporting the issue and put a patch to fix it. I do see the similar log before and huge number of warning messages are really annoying. +1 on refactor some log aggregation roll monitoring configuration logic from AppLogAggregatorImpl to LogAggregationService. It should belongs share to all AppAggregators rather than a particular AppAggregator. That should get rid of annoying message like: "rollingMonitorInterval is set as -1". However, for another annoying messages on permission issue - "Remote Root Log Dir [/app-logs] already exist, but with incorrect permissions.". It will still exist after the patch as verifyAndCreateRemoteLogDir() get called every time when we init an app. We should figure some way to get rid of it and warn (or even error) only one time for permission issue on remoteRootLogDir.

Hadoop QA added a comment - 19/May/16 16:48 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 28s trunk passed +1 compile 0m 29s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 34s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 54s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed -1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 6 new + 193 unchanged - 5 fixed = 199 total (was 198) +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 0s the patch passed +1 javadoc 0m 22s the patch passed +1 unit 12m 3s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 28m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804834/YARN-5112.patch JIRA Issue YARN-5112 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a73415ac96a2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 141873c Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11556/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11556/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11556/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

Jian He added a comment - 19/May/16 17:18 Thanks for the review, Junping. Forgot to remove verifyAndCreateRemoteLogDir from the old caller, updated the patch.

Junping Du added a comment - 19/May/16 21:47 Thanks Jian He for updating the patch. Remove verifyAndCreateRemoteLogDir() from initApp() could have two issues: YarnRuntimeException could be thrown without proper handling that will crash LogAggregationService/ContainerManager/NM. If root directory is deleted by admin/tools unintentionally, then new launched apps after the mistaking operation have no chance to fix it. I think the safe way is tracking down the permission issue and only log for the first time we hit or the 1st time after permission back to normal for a while. What do you think?

Jian He added a comment - 19/May/16 22:08 Thanks for the review, sounds good to me. Uploaded a new patch to address the comments.

Hadoop QA added a comment - 19/May/16 22:48 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 6s trunk passed +1 compile 0m 26s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 11s trunk passed +1 findbugs 0m 44s trunk passed +1 javadoc 0m 19s trunk passed +1 mvninstall 0m 23s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed -1 checkstyle 0m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 6 new + 36 unchanged - 5 fixed = 42 total (was 41) +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 51s the patch passed +1 javadoc 0m 17s the patch passed +1 unit 11m 20s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 15s Patch does not generate ASF License warnings. 24m 35s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805066/YARN-5112.3.patch JIRA Issue YARN-5112 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e7eb7553c1b5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 22fcd81 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11572/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11572/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11572/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

Junping Du added a comment - 19/May/16 23:54 Thanks Jian for the patch. Latest patch LGTM. However, can we add a unit test in TestLogAggregationService to verify that when root dir permission issue happens, only 1 warn message there?

Jian He added a comment - 20/May/16 00:13 Fixing the checkstyle, it's no straightforward to test the log message occurence, given it's just a log message, i think no unit test is fine..

Hadoop QA added a comment - 20/May/16 00:38 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 16s trunk passed +1 compile 0m 29s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 45s trunk passed +1 javadoc 0m 19s trunk passed +1 mvninstall 0m 23s the patch passed +1 compile 0m 25s the patch passed +1 javac 0m 25s the patch passed -1 checkstyle 0m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 3 new + 37 unchanged - 5 fixed = 40 total (was 42) +1 mvnsite 0m 28s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 53s the patch passed +1 javadoc 0m 17s the patch passed +1 unit 11m 32s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 15s Patch does not generate ASF License warnings. 25m 15s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805083/YARN-5112.4.patch JIRA Issue YARN-5112 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6afb856bc630 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 42c22f7 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11575/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11575/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11575/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

Hadoop QA added a comment - 20/May/16 01:08 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s YARN-5112 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805101/YARN-5112.5.patch JIRA Issue YARN-5112 Console output https://builds.apache.org/job/PreCommit-YARN-Build/11579/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

Junping Du added a comment - 20/May/16 01:33 it's no straightforward to test the log message occurence, given it's just a log message, i think no unit test is fine. Sure. Then only fix checkstyle issue should be fine to me.

Hadoop QA added a comment - 20/May/16 02:52 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 15s trunk passed +1 compile 0m 25s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 38s trunk passed +1 javadoc 0m 18s trunk passed +1 mvninstall 0m 21s the patch passed +1 compile 0m 21s the patch passed +1 javac 0m 21s the patch passed -1 checkstyle 0m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 2 new + 36 unchanged - 5 fixed = 38 total (was 41) +1 mvnsite 0m 22s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 43s the patch passed +1 javadoc 0m 16s the patch passed +1 unit 11m 41s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 23m 57s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805111/YARN-5112.6.patch JIRA Issue YARN-5112 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6dc9b231bf27 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 21d2b90 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11582/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11582/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11582/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

Junping Du added a comment - 20/May/16 12:22 Thanks Jian He for updating the patch. The left two checkstyle issues is not fixable and let's keep it there. +1 on latest patch. Will commit it shortly if no further comments from others.

Junping Du added a comment - 22/May/16 23:03 I have commit the patch to trunk and branch-2. Thanks Jian He for patch contribution!

Hudson added a comment - 22/May/16 23:14 SUCCESS: Integrated in Hadoop-trunk-Commit #9840 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9840/ ) YARN-5112 . Excessive log warnings for directory permission issue on NM (junping_du: rev 6161d9ba5230f553db5f5490dce67e2afd1e29ca) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java