Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5112

Excessive log warnings for directory permission issue on NM recovery.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When there are a lot of apps to recover in NM store, NM prints these two lines for each app, which gets annoying.

      2015-10-13 01:58:40,277 WARN  logaggregation.LogAggregationService (LogAggregationService.java:verifyAndCreateRemoteLogDir(195)) - Remote Root Log Dir [/app-logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple users.
      1111336 2015-10-13 01:58:40,277 WARN  logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:<init>(182)) - rollingMonitorInterval is set as -1. The log rolling mornitoring interval is disabled. The logs will be aggregated after this application is finished.
      
      1. YARN-5112.patch
        9 kB
        Jian He
      2. YARN-5112.6.patch
        9 kB
        Jian He
      3. YARN-5112.5.patch
        1 kB
        Jian He
      4. YARN-5112.4.patch
        8 kB
        Jian He
      5. YARN-5112.3.patch
        8 kB
        Jian He
      6. YARN-5112.2.patch
        9 kB
        Jian He

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #9840 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9840/)
        YARN-5112. Excessive log warnings for directory permission issue on NM (junping_du: rev 6161d9ba5230f553db5f5490dce67e2afd1e29ca)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9840 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9840/ ) YARN-5112 . Excessive log warnings for directory permission issue on NM (junping_du: rev 6161d9ba5230f553db5f5490dce67e2afd1e29ca) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java
        Hide
        djp Junping Du added a comment -

        I have commit the patch to trunk and branch-2. Thanks Jian He for patch contribution!

        Show
        djp Junping Du added a comment - I have commit the patch to trunk and branch-2. Thanks Jian He for patch contribution!
        Hide
        djp Junping Du added a comment -

        Thanks Jian He for updating the patch. The left two checkstyle issues is not fixable and let's keep it there.
        +1 on latest patch. Will commit it shortly if no further comments from others.

        Show
        djp Junping Du added a comment - Thanks Jian He for updating the patch. The left two checkstyle issues is not fixable and let's keep it there. +1 on latest patch. Will commit it shortly if no further comments from others.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 15s trunk passed
        +1 compile 0m 25s trunk passed
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 0m 29s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 38s trunk passed
        +1 javadoc 0m 18s trunk passed
        +1 mvninstall 0m 21s the patch passed
        +1 compile 0m 21s the patch passed
        +1 javac 0m 21s the patch passed
        -1 checkstyle 0m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 2 new + 36 unchanged - 5 fixed = 38 total (was 41)
        +1 mvnsite 0m 22s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 43s the patch passed
        +1 javadoc 0m 16s the patch passed
        +1 unit 11m 41s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 18s Patch does not generate ASF License warnings.
        23m 57s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805111/YARN-5112.6.patch
        JIRA Issue YARN-5112
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 6dc9b231bf27 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 21d2b90
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11582/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11582/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11582/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 15s trunk passed +1 compile 0m 25s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 38s trunk passed +1 javadoc 0m 18s trunk passed +1 mvninstall 0m 21s the patch passed +1 compile 0m 21s the patch passed +1 javac 0m 21s the patch passed -1 checkstyle 0m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 2 new + 36 unchanged - 5 fixed = 38 total (was 41) +1 mvnsite 0m 22s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 43s the patch passed +1 javadoc 0m 16s the patch passed +1 unit 11m 41s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 23m 57s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805111/YARN-5112.6.patch JIRA Issue YARN-5112 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6dc9b231bf27 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 21d2b90 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11582/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11582/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11582/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        djp Junping Du added a comment -

        it's no straightforward to test the log message occurence, given it's just a log message, i think no unit test is fine.

        Sure. Then only fix checkstyle issue should be fine to me.

        Show
        djp Junping Du added a comment - it's no straightforward to test the log message occurence, given it's just a log message, i think no unit test is fine. Sure. Then only fix checkstyle issue should be fine to me.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 4s YARN-5112 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805101/YARN-5112.5.patch
        JIRA Issue YARN-5112
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11579/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s YARN-5112 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805101/YARN-5112.5.patch JIRA Issue YARN-5112 Console output https://builds.apache.org/job/PreCommit-YARN-Build/11579/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 16s trunk passed
        +1 compile 0m 29s trunk passed
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 29s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 0m 45s trunk passed
        +1 javadoc 0m 19s trunk passed
        +1 mvninstall 0m 23s the patch passed
        +1 compile 0m 25s the patch passed
        +1 javac 0m 25s the patch passed
        -1 checkstyle 0m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 3 new + 37 unchanged - 5 fixed = 40 total (was 42)
        +1 mvnsite 0m 28s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 53s the patch passed
        +1 javadoc 0m 17s the patch passed
        +1 unit 11m 32s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 15s Patch does not generate ASF License warnings.
        25m 15s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805083/YARN-5112.4.patch
        JIRA Issue YARN-5112
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 6afb856bc630 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 42c22f7
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11575/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11575/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11575/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 16s trunk passed +1 compile 0m 29s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 45s trunk passed +1 javadoc 0m 19s trunk passed +1 mvninstall 0m 23s the patch passed +1 compile 0m 25s the patch passed +1 javac 0m 25s the patch passed -1 checkstyle 0m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 3 new + 37 unchanged - 5 fixed = 40 total (was 42) +1 mvnsite 0m 28s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 53s the patch passed +1 javadoc 0m 17s the patch passed +1 unit 11m 32s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 15s Patch does not generate ASF License warnings. 25m 15s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805083/YARN-5112.4.patch JIRA Issue YARN-5112 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6afb856bc630 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 42c22f7 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11575/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11575/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11575/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jianhe Jian He added a comment -

        Fixing the checkstyle, it's no straightforward to test the log message occurence, given it's just a log message, i think no unit test is fine..

        Show
        jianhe Jian He added a comment - Fixing the checkstyle, it's no straightforward to test the log message occurence, given it's just a log message, i think no unit test is fine..
        Hide
        djp Junping Du added a comment -

        Thanks Jian for the patch. Latest patch LGTM. However, can we add a unit test in TestLogAggregationService to verify that when root dir permission issue happens, only 1 warn message there?

        Show
        djp Junping Du added a comment - Thanks Jian for the patch. Latest patch LGTM. However, can we add a unit test in TestLogAggregationService to verify that when root dir permission issue happens, only 1 warn message there?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 6s trunk passed
        +1 compile 0m 26s trunk passed
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 27s trunk passed
        +1 mvneclipse 0m 11s trunk passed
        +1 findbugs 0m 44s trunk passed
        +1 javadoc 0m 19s trunk passed
        +1 mvninstall 0m 23s the patch passed
        +1 compile 0m 24s the patch passed
        +1 javac 0m 24s the patch passed
        -1 checkstyle 0m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 6 new + 36 unchanged - 5 fixed = 42 total (was 41)
        +1 mvnsite 0m 25s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 51s the patch passed
        +1 javadoc 0m 17s the patch passed
        +1 unit 11m 20s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 15s Patch does not generate ASF License warnings.
        24m 35s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805066/YARN-5112.3.patch
        JIRA Issue YARN-5112
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e7eb7553c1b5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 22fcd81
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11572/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11572/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11572/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 6s trunk passed +1 compile 0m 26s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 11s trunk passed +1 findbugs 0m 44s trunk passed +1 javadoc 0m 19s trunk passed +1 mvninstall 0m 23s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed -1 checkstyle 0m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 6 new + 36 unchanged - 5 fixed = 42 total (was 41) +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 51s the patch passed +1 javadoc 0m 17s the patch passed +1 unit 11m 20s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 15s Patch does not generate ASF License warnings. 24m 35s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805066/YARN-5112.3.patch JIRA Issue YARN-5112 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e7eb7553c1b5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 22fcd81 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11572/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11572/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11572/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jianhe Jian He added a comment -

        Thanks for the review, sounds good to me.
        Uploaded a new patch to address the comments.

        Show
        jianhe Jian He added a comment - Thanks for the review, sounds good to me. Uploaded a new patch to address the comments.
        Hide
        djp Junping Du added a comment -

        Thanks Jian He for updating the patch. Remove verifyAndCreateRemoteLogDir() from initApp() could have two issues:

        • YarnRuntimeException could be thrown without proper handling that will crash LogAggregationService/ContainerManager/NM.
        • If root directory is deleted by admin/tools unintentionally, then new launched apps after the mistaking operation have no chance to fix it.
          I think the safe way is tracking down the permission issue and only log for the first time we hit or the 1st time after permission back to normal for a while. What do you think?
        Show
        djp Junping Du added a comment - Thanks Jian He for updating the patch. Remove verifyAndCreateRemoteLogDir() from initApp() could have two issues: YarnRuntimeException could be thrown without proper handling that will crash LogAggregationService/ContainerManager/NM. If root directory is deleted by admin/tools unintentionally, then new launched apps after the mistaking operation have no chance to fix it. I think the safe way is tracking down the permission issue and only log for the first time we hit or the 1st time after permission back to normal for a while. What do you think?
        Hide
        jianhe Jian He added a comment -

        Thanks for the review, Junping. Forgot to remove verifyAndCreateRemoteLogDir from the old caller, updated the patch.

        Show
        jianhe Jian He added a comment - Thanks for the review, Junping. Forgot to remove verifyAndCreateRemoteLogDir from the old caller, updated the patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 28s trunk passed
        +1 compile 0m 29s trunk passed
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 34s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 0m 54s trunk passed
        +1 javadoc 0m 22s trunk passed
        +1 mvninstall 0m 30s the patch passed
        +1 compile 0m 31s the patch passed
        +1 javac 0m 31s the patch passed
        -1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 6 new + 193 unchanged - 5 fixed = 199 total (was 198)
        +1 mvnsite 0m 33s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 0s the patch passed
        +1 javadoc 0m 22s the patch passed
        +1 unit 12m 3s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 19s Patch does not generate ASF License warnings.
        28m 2s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804834/YARN-5112.patch
        JIRA Issue YARN-5112
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a73415ac96a2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 141873c
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11556/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11556/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11556/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 28s trunk passed +1 compile 0m 29s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 34s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 54s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed -1 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 6 new + 193 unchanged - 5 fixed = 199 total (was 198) +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 0s the patch passed +1 javadoc 0m 22s the patch passed +1 unit 12m 3s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 28m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804834/YARN-5112.patch JIRA Issue YARN-5112 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a73415ac96a2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 141873c Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11556/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11556/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11556/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        djp Junping Du added a comment -

        Thanks Jian He for reporting the issue and put a patch to fix it. I do see the similar log before and huge number of warning messages are really annoying.
        +1 on refactor some log aggregation roll monitoring configuration logic from AppLogAggregatorImpl to LogAggregationService. It should belongs share to all AppAggregators rather than a particular AppAggregator. That should get rid of annoying message like: "rollingMonitorInterval is set as -1".
        However, for another annoying messages on permission issue - "Remote Root Log Dir [/app-logs] already exist, but with incorrect permissions.". It will still exist after the patch as verifyAndCreateRemoteLogDir() get called every time when we init an app. We should figure some way to get rid of it and warn (or even error) only one time for permission issue on remoteRootLogDir.

        Show
        djp Junping Du added a comment - Thanks Jian He for reporting the issue and put a patch to fix it. I do see the similar log before and huge number of warning messages are really annoying. +1 on refactor some log aggregation roll monitoring configuration logic from AppLogAggregatorImpl to LogAggregationService. It should belongs share to all AppAggregators rather than a particular AppAggregator. That should get rid of annoying message like: "rollingMonitorInterval is set as -1". However, for another annoying messages on permission issue - "Remote Root Log Dir [/app-logs] already exist, but with incorrect permissions.". It will still exist after the patch as verifyAndCreateRemoteLogDir() get called every time when we init an app. We should figure some way to get rid of it and warn (or even error) only one time for permission issue on remoteRootLogDir.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 35s trunk passed
        +1 compile 0m 28s trunk passed
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 28s trunk passed
        +1 mvneclipse 0m 10s trunk passed
        +1 findbugs 0m 43s trunk passed
        +1 javadoc 0m 20s trunk passed
        +1 mvninstall 0m 23s the patch passed
        +1 compile 0m 24s the patch passed
        +1 javac 0m 24s the patch passed
        -1 checkstyle 0m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 6 new + 193 unchanged - 5 fixed = 199 total (was 198)
        +1 mvnsite 0m 23s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 53s the patch passed
        +1 javadoc 0m 16s the patch passed
        -1 unit 11m 28s hadoop-yarn-server-nodemanager in the patch failed.
        +1 asflicense 0m 15s Patch does not generate ASF License warnings.
        26m 16s



        Reason Tests
        Failed junit tests hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804834/YARN-5112.patch
        JIRA Issue YARN-5112
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f3ecde06b25a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1597630
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11547/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11547/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11547/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11547/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11547/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 35s trunk passed +1 compile 0m 28s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 10s trunk passed +1 findbugs 0m 43s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 23s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed -1 checkstyle 0m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 6 new + 193 unchanged - 5 fixed = 199 total (was 198) +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 53s the patch passed +1 javadoc 0m 16s the patch passed -1 unit 11m 28s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 15s Patch does not generate ASF License warnings. 26m 16s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804834/YARN-5112.patch JIRA Issue YARN-5112 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f3ecde06b25a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1597630 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11547/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11547/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11547/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11547/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11547/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 45s trunk passed
        +1 compile 0m 25s trunk passed
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 27s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 0m 39s trunk passed
        +1 javadoc 0m 20s trunk passed
        +1 mvninstall 0m 27s the patch passed
        +1 compile 0m 24s the patch passed
        +1 javac 0m 24s the patch passed
        -1 checkstyle 0m 17s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 6 new + 194 unchanged - 5 fixed = 200 total (was 199)
        +1 mvnsite 0m 27s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 51s the patch passed
        +1 javadoc 0m 19s the patch passed
        -1 unit 11m 42s hadoop-yarn-server-nodemanager in the patch failed.
        +1 asflicense 0m 17s Patch does not generate ASF License warnings.
        24m 48s



        Reason Tests
        Failed junit tests hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804834/YARN-5112.patch
        JIRA Issue YARN-5112
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux d9c2eba187fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 010e6ac
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11536/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11536/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11536/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11536/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11536/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 45s trunk passed +1 compile 0m 25s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 39s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 27s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed -1 checkstyle 0m 17s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 6 new + 194 unchanged - 5 fixed = 200 total (was 199) +1 mvnsite 0m 27s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 51s the patch passed +1 javadoc 0m 19s the patch passed -1 unit 11m 42s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 24m 48s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804834/YARN-5112.patch JIRA Issue YARN-5112 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d9c2eba187fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 010e6ac Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11536/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11536/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11536/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11536/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11536/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jianhe Jian He added a comment -

        move the check and init part to the common LogAggregationService init.

        Show
        jianhe Jian He added a comment - move the check and init part to the common LogAggregationService init.

          People

          • Assignee:
            jianhe Jian He
            Reporter:
            jianhe Jian He
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development