Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4773

Log aggregation performs extraneous filesystem operations when rolling log aggregation is disabled

    Details

    • Hadoop Flags:
      Reviewed

      Description

      I noticed when log aggregation occurs for an application the nodemanager is listing the application's log directory in HDFS. Apparently this is for removing old logs before uploading new ones. This is a wasteful operation when rolling log aggregation is disabled, since there will be no prior logs in HDFS – aggregation only occurs once when rolling log aggregation is disabled.

      1. YARN-4773-branch-2.6.patch
        1 kB
        Jun Gong
      2. YARN-4773.03.patch
        5 kB
        Jun Gong
      3. YARN-4773.02.patch
        5 kB
        Jun Gong
      4. YARN-4773.01.patch
        5 kB
        Jun Gong

        Activity

        Hide
        mingma Ming Ma added a comment -

        Jason Lowe, can you please confirm if it has been fixed by YARN-4720 which has removed the extra remoteFS.exists(remoteNodeTmpLogFileForApp) for all log aggregation scenarios.

        Show
        mingma Ming Ma added a comment - Jason Lowe , can you please confirm if it has been fixed by YARN-4720 which has removed the extra remoteFS.exists(remoteNodeTmpLogFileForApp) for all log aggregation scenarios.
        Hide
        jlowe Jason Lowe added a comment -

        It is not fixed by YARN-4720 since this is a listStatus call not a getFileStatus call. Specifically the call is coming from AppLogAggregatorImpl#cleanOldLogs where it does a listStatus looking for old logs to remove. If rolling log aggregation is disabled then the listStatus call per node is unnecessary. We would only be uploading logs when the application completes, and there should be no old logs to remove in that scenario.

        Show
        jlowe Jason Lowe added a comment - It is not fixed by YARN-4720 since this is a listStatus call not a getFileStatus call. Specifically the call is coming from AppLogAggregatorImpl#cleanOldLogs where it does a listStatus looking for old logs to remove. If rolling log aggregation is disabled then the listStatus call per node is unnecessary. We would only be uploading logs when the application completes, and there should be no old logs to remove in that scenario.
        Hide
        hex108 Jun Gong added a comment -

        Thanks Jason Lowe for reporting the issue.

        YARN-4720 tries to skip unnecessary NN operations for every call AppLogAggregatorImpl#uploadLogsForContainers if pendingContainerInThisCycle is empty. IIUC Jason Lowe means this case: rolling log aggregation is disabled, when app completes, we call AppLogAggregatorImpl#uploadLogsForContainers(true), and pendingContainerInThisCycle is not empty, then we will call AppLogAggregatorImpl#cleanOldLogs, however we do not need call AppLogAggregatorImpl#cleanOldLogs because there have been no containers' logs uploaded before.

        Show
        hex108 Jun Gong added a comment - Thanks Jason Lowe for reporting the issue. YARN-4720 tries to skip unnecessary NN operations for every call AppLogAggregatorImpl#uploadLogsForContainers if pendingContainerInThisCycle is empty. IIUC Jason Lowe means this case: rolling log aggregation is disabled, when app completes, we call AppLogAggregatorImpl#uploadLogsForContainers(true) , and pendingContainerInThisCycle is not empty, then we will call AppLogAggregatorImpl#cleanOldLogs , however we do not need call AppLogAggregatorImpl#cleanOldLogs because there have been no containers' logs uploaded before.
        Hide
        jlowe Jason Lowe added a comment -

        Yes, Jun Gong that is the scenario.

        Show
        jlowe Jason Lowe added a comment - Yes, Jun Gong that is the scenario.
        Hide
        hex108 Jun Gong added a comment -

        Thanks Jason Lowe for confirming it. I will attach a patch to fix it if you are not working on it.

        Show
        hex108 Jun Gong added a comment - Thanks Jason Lowe for confirming it. I will attach a patch to fix it if you are not working on it.
        Hide
        jlowe Jason Lowe added a comment -

        Feel free, as I'm currently busy with other tasks. I filed it and left it unassigned in the hope someone in the community would have the time to pick it up, so thanks!

        Show
        jlowe Jason Lowe added a comment - Feel free, as I'm currently busy with other tasks. I filed it and left it unassigned in the hope someone in the community would have the time to pick it up, so thanks!
        Hide
        hex108 Jun Gong added a comment -

        Attach a patch.

        If logAggregationInRolling is false, it means rolling log aggregation is disabled, then we could avoid calling AppLogAggregatorImpl#cleanOldLogs.

        Show
        hex108 Jun Gong added a comment - Attach a patch. If logAggregationInRolling is false, it means rolling log aggregation is disabled, then we could avoid calling AppLogAggregatorImpl#cleanOldLogs .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 39s trunk passed
        +1 compile 0m 22s trunk passed with JDK v1.8.0_74
        +1 compile 0m 25s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 27s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 50s trunk passed
        +1 javadoc 0m 18s trunk passed with JDK v1.8.0_74
        +1 javadoc 0m 21s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 24s the patch passed
        +1 compile 0m 19s the patch passed with JDK v1.8.0_74
        +1 javac 0m 19s the patch passed
        +1 compile 0m 24s the patch passed with JDK v1.7.0_95
        +1 javac 0m 24s the patch passed
        -1 checkstyle 0m 13s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 1 new + 17 unchanged - 1 fixed = 18 total (was 18)
        +1 mvnsite 0m 28s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 59s the patch passed
        +1 javadoc 0m 16s the patch passed with JDK v1.8.0_74
        +1 javadoc 0m 20s the patch passed with JDK v1.7.0_95
        +1 unit 9m 7s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74.
        +1 unit 9m 43s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 19s Patch does not generate ASF License warnings.
        33m 49s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792516/YARN-4773.01.patch
        JIRA Issue YARN-4773
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 483af54f920e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 318c9b6
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10754/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10754/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10754/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 39s trunk passed +1 compile 0m 22s trunk passed with JDK v1.8.0_74 +1 compile 0m 25s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 50s trunk passed +1 javadoc 0m 18s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 21s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 24s the patch passed +1 compile 0m 19s the patch passed with JDK v1.8.0_74 +1 javac 0m 19s the patch passed +1 compile 0m 24s the patch passed with JDK v1.7.0_95 +1 javac 0m 24s the patch passed -1 checkstyle 0m 13s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 1 new + 17 unchanged - 1 fixed = 18 total (was 18) +1 mvnsite 0m 28s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 59s the patch passed +1 javadoc 0m 16s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 20s the patch passed with JDK v1.7.0_95 +1 unit 9m 7s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74. +1 unit 9m 43s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 33m 49s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12792516/YARN-4773.01.patch JIRA Issue YARN-4773 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 483af54f920e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 318c9b6 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10754/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10754/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10754/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Jun Gong/Jason Lowe

        however we do not need call AppLogAggregatorImpl#cleanOldLogs because there have been no containers' logs uploaded before.

        While aggregation was going on NM got killed and restarted again .Will thr be a possiblity for <HOSTNAME>.tmp file exists in folder??

        Show
        bibinchundatt Bibin A Chundatt added a comment - Jun Gong / Jason Lowe however we do not need call AppLogAggregatorImpl#cleanOldLogs because there have been no containers' logs uploaded before. While aggregation was going on NM got killed and restarted again .Will thr be a possiblity for <HOSTNAME>.tmp file exists in folder??
        Hide
        hex108 Jun Gong added a comment -

        hi Bibin A Chundatt, if the aggregation restarts again, previous tmp file will be overridden because tmp file name does not change.

        Show
        hex108 Jun Gong added a comment - hi Bibin A Chundatt , if the aggregation restarts again, previous tmp file will be overridden because tmp file name does not change.
        Hide
        jlowe Jason Lowe added a comment -

        Patch looks good overall, just a few nits:

        • getCleanupOldLogTimes should be package private, eliminating the need for the Private annotation
        • Why was stdout changed to sysout in the test? Seems unrelated.
        Show
        jlowe Jason Lowe added a comment - Patch looks good overall, just a few nits: getCleanupOldLogTimes should be package private, eliminating the need for the Private annotation Why was stdout changed to sysout in the test? Seems unrelated.
        Hide
        hex108 Jun Gong added a comment -

        Thanks Jason Lowe for the review and comments.

        Why was stdout changed to sysout in the test? Seems unrelated.

        In the test case, testSkipUnnecessaryNNOperationsForService calls setRolledLogsIncludePattern("sysout"), if generated container log is "stdout", there will be no container logs that needs uploaded, then uploadedLogsInThisCycle will always be false. In order to make both uploadedLogsInThisCycle and logAggregationInRolling be true for some cases, I change "stdout" to "sysout", then we could see cases that cleanOldLogs is called.

        Attached a new patch to address the first comment.

        Show
        hex108 Jun Gong added a comment - Thanks Jason Lowe for the review and comments. Why was stdout changed to sysout in the test? Seems unrelated. In the test case, testSkipUnnecessaryNNOperationsForService calls setRolledLogsIncludePattern("sysout") , if generated container log is "stdout", there will be no container logs that needs uploaded, then uploadedLogsInThisCycle will always be false. In order to make both uploadedLogsInThisCycle and logAggregationInRolling be true for some cases, I change "stdout" to "sysout", then we could see cases that cleanOldLogs is called. Attached a new patch to address the first comment.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 30s trunk passed
        +1 compile 0m 28s trunk passed with JDK v1.8.0_74
        +1 compile 0m 27s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 29s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 0m 52s trunk passed
        +1 javadoc 0m 23s trunk passed with JDK v1.8.0_74
        +1 javadoc 0m 24s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 27s the patch passed
        +1 compile 0m 28s the patch passed with JDK v1.8.0_74
        +1 javac 0m 28s the patch passed
        +1 compile 0m 26s the patch passed with JDK v1.7.0_95
        +1 javac 0m 26s the patch passed
        -1 checkstyle 0m 13s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 1 new + 17 unchanged - 1 fixed = 18 total (was 18)
        +1 mvnsite 0m 29s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 4s the patch passed
        +1 javadoc 0m 21s the patch passed with JDK v1.8.0_74
        +1 javadoc 0m 20s the patch passed with JDK v1.7.0_95
        +1 unit 9m 59s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74.
        +1 unit 9m 58s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 20s Patch does not generate ASF License warnings.
        36m 37s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795348/YARN-4773.02.patch
        JIRA Issue YARN-4773
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 53b56bfad159 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2e1d0ff
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10874/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10874/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10874/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 30s trunk passed +1 compile 0m 28s trunk passed with JDK v1.8.0_74 +1 compile 0m 27s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 52s trunk passed +1 javadoc 0m 23s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 24s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 27s the patch passed +1 compile 0m 28s the patch passed with JDK v1.8.0_74 +1 javac 0m 28s the patch passed +1 compile 0m 26s the patch passed with JDK v1.7.0_95 +1 javac 0m 26s the patch passed -1 checkstyle 0m 13s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 1 new + 17 unchanged - 1 fixed = 18 total (was 18) +1 mvnsite 0m 29s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 4s the patch passed +1 javadoc 0m 21s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 20s the patch passed with JDK v1.7.0_95 +1 unit 9m 59s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74. +1 unit 9m 58s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 36m 37s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795348/YARN-4773.02.patch JIRA Issue YARN-4773 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 53b56bfad159 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2e1d0ff Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10874/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10874/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10874/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks for updating the patch!

        I see the Private annotation was removed, but getCleanupOldLogTimes was not made package-private.

        Show
        jlowe Jason Lowe added a comment - Thanks for updating the patch! I see the Private annotation was removed, but getCleanupOldLogTimes was not made package-private.
        Hide
        hex108 Jun Gong added a comment -

        Sorry... Attached a new patch.

        Show
        hex108 Jun Gong added a comment - Sorry... Attached a new patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 46s trunk passed
        +1 compile 0m 23s trunk passed with JDK v1.8.0_74
        +1 compile 0m 26s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 28s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 0m 52s trunk passed
        +1 javadoc 0m 18s trunk passed with JDK v1.8.0_74
        +1 javadoc 0m 22s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 25s the patch passed
        +1 compile 0m 20s the patch passed with JDK v1.8.0_74
        +1 javac 0m 20s the patch passed
        +1 compile 0m 25s the patch passed with JDK v1.7.0_95
        +1 javac 0m 25s the patch passed
        -1 checkstyle 0m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 1 new + 17 unchanged - 1 fixed = 18 total (was 18)
        +1 mvnsite 0m 27s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 58s the patch passed
        +1 javadoc 0m 15s the patch passed with JDK v1.8.0_74
        +1 javadoc 0m 19s the patch passed with JDK v1.7.0_95
        +1 unit 9m 10s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74.
        +1 unit 9m 39s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95.
        -1 asflicense 0m 17s Patch generated 1 ASF License warnings.
        33m 59s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795531/YARN-4773.03.patch
        JIRA Issue YARN-4773
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8428de73e660 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / a337ceb
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10885/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10885/testReport/
        asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10885/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10885/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 46s trunk passed +1 compile 0m 23s trunk passed with JDK v1.8.0_74 +1 compile 0m 26s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 52s trunk passed +1 javadoc 0m 18s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 22s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 25s the patch passed +1 compile 0m 20s the patch passed with JDK v1.8.0_74 +1 javac 0m 20s the patch passed +1 compile 0m 25s the patch passed with JDK v1.7.0_95 +1 javac 0m 25s the patch passed -1 checkstyle 0m 14s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: patch generated 1 new + 17 unchanged - 1 fixed = 18 total (was 18) +1 mvnsite 0m 27s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 58s the patch passed +1 javadoc 0m 15s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 19s the patch passed with JDK v1.7.0_95 +1 unit 9m 10s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_74. +1 unit 9m 39s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_95. -1 asflicense 0m 17s Patch generated 1 ASF License warnings. 33m 59s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795531/YARN-4773.03.patch JIRA Issue YARN-4773 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8428de73e660 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a337ceb Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/10885/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10885/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/10885/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/10885/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jlowe Jason Lowe added a comment -

        +1 committing this.

        Show
        jlowe Jason Lowe added a comment - +1 committing this.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks, Jun Gong! I committed this to trunk, branch-2, and branch-2.8.

        It does not apply cleanly to 2.7 or 2.6, would you mind putting up a patch for those as well?

        Show
        jlowe Jason Lowe added a comment - Thanks, Jun Gong ! I committed this to trunk, branch-2, and branch-2.8. It does not apply cleanly to 2.7 or 2.6, would you mind putting up a patch for those as well?
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9515 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9515/)
        YARN-4773. Log aggregation performs extraneous filesystem operations (jlowe: rev 948b75807068c304ffe789e32f2b850c0d653e0a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9515 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9515/ ) YARN-4773 . Log aggregation performs extraneous filesystem operations (jlowe: rev 948b75807068c304ffe789e32f2b850c0d653e0a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java
        Hide
        hex108 Jun Gong added a comment -

        Thanks Jason Lowe for review, suggestion and commit!

        The test case in the patch depends on YARN-4720 which is not backported to branch 2.6 and 2.7, so I removed the test case, and updated a new patch YARN-4773.branch-2.6.patch for 2.6 and 2.7. Is it OK?

        Show
        hex108 Jun Gong added a comment - Thanks Jason Lowe for review, suggestion and commit! The test case in the patch depends on YARN-4720 which is not backported to branch 2.6 and 2.7, so I removed the test case, and updated a new patch YARN-4773 .branch-2.6.patch for 2.6 and 2.7. Is it OK?
        Hide
        jlowe Jason Lowe added a comment -

        Apologies for the long delay.

        +1 for the branch-2.6 patch. Thanks, Jun! I committed it to branch-2.7 and branch-2.6.

        Show
        jlowe Jason Lowe added a comment - Apologies for the long delay. +1 for the branch-2.6 patch. Thanks, Jun! I committed it to branch-2.7 and branch-2.6.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

          People

          • Assignee:
            hex108 Jun Gong
            Reporter:
            jlowe Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development