Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5700

historyServer can't show container's log when aggregation is not enabled

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.23.7, 2.0.4-alpha, 2.2.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      yarn.log-aggregation-enable=false , HistoryServer will show like this:

      Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669

      Description

      When yarn.log-aggregation-enable is seted to false, after a MR_App complete, we can't view the container's log from the HistoryServer, it shows message like:
      Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669

      Since we don't want to aggregate the container's log, because it will be a pressure to namenode. but sometimes we also want to take a look at container's log.
      Should we show the container's log across HistoryServer even if yarn.log-aggregation-enable is seted to false.

      1. yarn-647.patch
        3 kB
        Hong Shen
      2. yarn-647-2.patch
        3 kB
        Hong Shen

        Activity

        Hide
        Hong Shen added a comment -

        add a patch

        Show
        Hong Shen added a comment - add a patch
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12581885/yarn-647.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2390//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581885/yarn-647.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2390//console This message is automatically generated.
        Hide
        Zhijie Shen added a comment -
        +        if (!aggregation) {
        +          logsLink = join(HttpConfig.getSchemePrefix(), nodeHttpAddr, 
        +              "/node", "/containerlogs/",
        +              containerIdString, "/", app.getJob().getUserName());
        

        I'm afraid this link will not work as well. When a container is stopped, NM local logs will be deleted and not be accessible via webUI or service again.

        However, I'm afraid "Try the nodemanager at ..." is a bit misleading as well, which makes users think that the logs are available via NM webUI. The fact is that for debugging purpose, we can config "yarn.nodemanager.delete.debug-delay-sec" to delay the deletion of NM local logs, but it's not user-oriented, and anyway, they're not accessible via web. Maybe we want to remove misleading words here?

        Show
        Zhijie Shen added a comment - + if (!aggregation) { + logsLink = join(HttpConfig.getSchemePrefix(), nodeHttpAddr, + "/node" , "/containerlogs/" , + containerIdString, "/" , app.getJob().getUserName()); I'm afraid this link will not work as well. When a container is stopped, NM local logs will be deleted and not be accessible via webUI or service again. However, I'm afraid "Try the nodemanager at ..." is a bit misleading as well, which makes users think that the logs are available via NM webUI. The fact is that for debugging purpose, we can config "yarn.nodemanager.delete.debug-delay-sec" to delay the deletion of NM local logs, but it's not user-oriented, and anyway, they're not accessible via web. Maybe we want to remove misleading words here?
        Hide
        caolong added a comment -

        my hadoop cluster set yarn.log-aggregation-enable=false and yarn.nodemanager.log.retain-seconds=259200
        so the log remain 3days in nn。
        so i need view logs by historyserver .

        Show
        caolong added a comment - my hadoop cluster set yarn.log-aggregation-enable=false and yarn.nodemanager.log.retain-seconds=259200 so the log remain 3days in nn。 so i need view logs by historyserver .
        Hide
        Hong Shen added a comment -

        Thanks Zhijie!
        Like caolong, we also set yarn.nodemanager.log.retain-seconds=259200, so NM local logs won't be deleted after container stop,
        I think if yarn.log-aggregation-enable=false and yarn.nodemanager.log.retain-seconds>0, we can change the logsLink .

        Show
        Hong Shen added a comment - Thanks Zhijie! Like caolong, we also set yarn.nodemanager.log.retain-seconds=259200, so NM local logs won't be deleted after container stop, I think if yarn.log-aggregation-enable=false and yarn.nodemanager.log.retain-seconds>0, we can change the logsLink .
        Hide
        Hong Shen added a comment -

        add a new patch

        Show
        Hong Shen added a comment - add a new patch
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12620602/yarn-647-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/2738//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2738//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12620602/yarn-647-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/2738//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2738//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12620602/yarn-647-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4284//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4284//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12620602/yarn-647-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4284//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4284//console This message is automatically generated.

          People

          • Assignee:
            Hong Shen
            Reporter:
            Hong Shen
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development