Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5700

historyServer can't show container's log when aggregation is not enabled

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.23.7, 2.0.4-alpha, 2.2.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      yarn.log-aggregation-enable=false , HistoryServer will show like this:

      Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669

      Description

      When yarn.log-aggregation-enable is seted to false, after a MR_App complete, we can't view the container's log from the HistoryServer, it shows message like:
      Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669

      Since we don't want to aggregate the container's log, because it will be a pressure to namenode. but sometimes we also want to take a look at container's log.
      Should we show the container's log across HistoryServer even if yarn.log-aggregation-enable is seted to false.

      1. yarn-647-2.patch
        3 kB
        Hong Shen
      2. yarn-647.patch
        3 kB
        Hong Shen

        Activity

        Hong Shen created issue -
        Hong Shen made changes -
        Field Original Value New Value
        Description When yarn.log-aggregation-enable was seted to false, after a MR_App complete, we can't view the container's log from the HistoryServer, it shows message like:

        Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669

        Since we don't want to aggregate the container's log, because it will be a pressure to namenode. but sometimes we also want to take a look at container's log.
        Should we show the container's log across HistoryServer even if yarn.log-aggregation-enable was seted to false.
        Hong Shen made changes -
        Environment When yarn.log-aggregation-enable was seted to false, after a MR_App complete, we can't view the container's log from the HistoryServer, it shows message like:

        Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669

        Since we don't want to aggregate the container's log, because it will be a pressure to namenode. but sometimes we also want to take a look at container's log.

        Should we show the container's log across HistoryServer even if yarn.log-aggregation-enable was seted to false.
        When yarn.log-aggregation-enable was seted to false, after a MR_App complete, we can't view the container's log from the HistoryServer, it shows message like:

        Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669
        Hong Shen made changes -
        Description When yarn.log-aggregation-enable was seted to false, after a MR_App complete, we can't view the container's log from the HistoryServer, it shows message like:

        Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669

        Since we don't want to aggregate the container's log, because it will be a pressure to namenode. but sometimes we also want to take a look at container's log.
        Should we show the container's log across HistoryServer even if yarn.log-aggregation-enable was seted to false.


        Since we don't want to aggregate the container's log, because it will be a pressure to namenode. but sometimes we also want to take a look at container's log.
        Should we show the container's log across HistoryServer even if yarn.log-aggregation-enable was seted to false.
        Hong Shen made changes -
        Description

        Since we don't want to aggregate the container's log, because it will be a pressure to namenode. but sometimes we also want to take a look at container's log.
        Should we show the container's log across HistoryServer even if yarn.log-aggregation-enable was seted to false.
        When yarn.log-aggregation-enable was seted to false, after a MR_App complete, we can't view the container's log from the HistoryServer, it shows message like:
        Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669

        Since we don't want to aggregate the container's log, because it will be a pressure to namenode. but sometimes we also want to take a look at container's log.
        Should we show the container's log across HistoryServer even if yarn.log-aggregation-enable was seted to false.
        Hong Shen made changes -
        Environment When yarn.log-aggregation-enable was seted to false, after a MR_App complete, we can't view the container's log from the HistoryServer, it shows message like:

        Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669
         yarn.log-aggregation-enable=false

        Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669
        Hong Shen made changes -
        Environment  yarn.log-aggregation-enable=false

        Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669
         yarn.log-aggregation-enable=false , HistoryServer will show like this:

        Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669
        Hong Shen made changes -
        Summary historyServer can show container's log when aggregation is not enabled historyServer can't show container's log when aggregation is not enabled
        Hide
        Hong Shen added a comment -

        add a patch

        Show
        Hong Shen added a comment - add a patch
        Hong Shen made changes -
        Attachment yarn-647.patch [ 12581885 ]
        Hong Shen made changes -
        Description When yarn.log-aggregation-enable was seted to false, after a MR_App complete, we can't view the container's log from the HistoryServer, it shows message like:
        Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669

        Since we don't want to aggregate the container's log, because it will be a pressure to namenode. but sometimes we also want to take a look at container's log.
        Should we show the container's log across HistoryServer even if yarn.log-aggregation-enable was seted to false.
        When yarn.log-aggregation-enable is seted to false, after a MR_App complete, we can't view the container's log from the HistoryServer, it shows message like:
        Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669

        Since we don't want to aggregate the container's log, because it will be a pressure to namenode. but sometimes we also want to take a look at container's log.
        Should we show the container's log across HistoryServer even if yarn.log-aggregation-enable is seted to false.
        Liyin Liang made changes -
        Fix Version/s 2.2.1 [ 12325256 ]
        Liyin Liang made changes -
        Assignee shenhong [ shenhong ]
        Liyin Liang made changes -
        Component/s documentation [ 12320410 ]
        Liyin Liang made changes -
        Affects Version/s 2.2.0 [ 12325051 ]
        Liyin Liang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Target Version/s 2.2.1 [ 12325256 ]
        Fix Version/s 2.2.1 [ 12325256 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12581885/yarn-647.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2390//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581885/yarn-647.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2390//console This message is automatically generated.
        Hide
        Zhijie Shen added a comment -
        +        if (!aggregation) {
        +          logsLink = join(HttpConfig.getSchemePrefix(), nodeHttpAddr, 
        +              "/node", "/containerlogs/",
        +              containerIdString, "/", app.getJob().getUserName());
        

        I'm afraid this link will not work as well. When a container is stopped, NM local logs will be deleted and not be accessible via webUI or service again.

        However, I'm afraid "Try the nodemanager at ..." is a bit misleading as well, which makes users think that the logs are available via NM webUI. The fact is that for debugging purpose, we can config "yarn.nodemanager.delete.debug-delay-sec" to delay the deletion of NM local logs, but it's not user-oriented, and anyway, they're not accessible via web. Maybe we want to remove misleading words here?

        Show
        Zhijie Shen added a comment - + if (!aggregation) { + logsLink = join(HttpConfig.getSchemePrefix(), nodeHttpAddr, + "/node" , "/containerlogs/" , + containerIdString, "/" , app.getJob().getUserName()); I'm afraid this link will not work as well. When a container is stopped, NM local logs will be deleted and not be accessible via webUI or service again. However, I'm afraid "Try the nodemanager at ..." is a bit misleading as well, which makes users think that the logs are available via NM webUI. The fact is that for debugging purpose, we can config "yarn.nodemanager.delete.debug-delay-sec" to delay the deletion of NM local logs, but it's not user-oriented, and anyway, they're not accessible via web. Maybe we want to remove misleading words here?
        Zhijie Shen made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        caolong added a comment -

        my hadoop cluster set yarn.log-aggregation-enable=false and yarn.nodemanager.log.retain-seconds=259200
        so the log remain 3days in nn。
        so i need view logs by historyserver .

        Show
        caolong added a comment - my hadoop cluster set yarn.log-aggregation-enable=false and yarn.nodemanager.log.retain-seconds=259200 so the log remain 3days in nn。 so i need view logs by historyserver .
        caolong made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hong Shen added a comment -

        Thanks Zhijie!
        Like caolong, we also set yarn.nodemanager.log.retain-seconds=259200, so NM local logs won't be deleted after container stop,
        I think if yarn.log-aggregation-enable=false and yarn.nodemanager.log.retain-seconds>0, we can change the logsLink .

        Show
        Hong Shen added a comment - Thanks Zhijie! Like caolong, we also set yarn.nodemanager.log.retain-seconds=259200, so NM local logs won't be deleted after container stop, I think if yarn.log-aggregation-enable=false and yarn.nodemanager.log.retain-seconds>0, we can change the logsLink .
        Hide
        Hong Shen added a comment -

        add a new patch

        Show
        Hong Shen added a comment - add a new patch
        Hong Shen made changes -
        Attachment yarn-647-2.patch [ 12620602 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12620602/yarn-647-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/2738//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2738//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12620602/yarn-647-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/2738//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2738//console This message is automatically generated.
        Vinod Kumar Vavilapalli made changes -
        Project Hadoop YARN [ 12313722 ] Hadoop Map/Reduce [ 12310941 ]
        Key YARN-647 MAPREDUCE-5700
        Affects Version/s 2.2.0 [ 12325050 ]
        Affects Version/s 2.0.4-alpha [ 12324138 ]
        Affects Version/s 0.23.7 [ 12323954 ]
        Affects Version/s 0.23.7 [ 12323953 ]
        Affects Version/s 2.0.4-alpha [ 12324137 ]
        Affects Version/s 2.2.0 [ 12325051 ]
        Target Version/s 2.3.0 [ 12325256 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12620602/yarn-647-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4284//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4284//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12620602/yarn-647-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4284//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4284//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 34s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 35s There were no new javac warning messages.
        +1 javadoc 9m 53s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 26s The applied patch generated 4 new checkstyle issues (total was 41, now 44).
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 36s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 0m 43s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 mapreduce tests 5m 43s Tests passed in hadoop-mapreduce-client-hs.
            41m 28s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12620602/yarn-647-2.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 6ae2a0d
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5598/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-hs.txt
        whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5598/artifact/patchprocess/whitespace.txt
        hadoop-mapreduce-client-hs test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5598/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5598/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5598/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 34s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 53s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 26s The applied patch generated 4 new checkstyle issues (total was 41, now 44). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 0m 43s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 5m 43s Tests passed in hadoop-mapreduce-client-hs.     41m 28s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12620602/yarn-647-2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6ae2a0d checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5598/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-hs.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5598/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-hs test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5598/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5598/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5598/console This message was automatically generated.

          People

          • Assignee:
            Hong Shen
            Reporter:
            Hong Shen
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development