Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: log-aggregation
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Right now, we could specify applicationId, node http address and container ID to get the specify container log. Or we could only specify applicationId to get all the container logs. It is very hard for the users to get logs for AM container since the AMContainer logs have more useful information. Users need to know the AMContainer's container ID and related Node http address.

      We could improve the YARN Log Command to allow users to get AMContainer logs directly

      1. YARN-3347.4.patch
        39 kB
        Xuan Gong
      2. YARN-3347.3.1.patch
        33 kB
        Xuan Gong
      3. YARN-3347.3.rebase.patch
        33 kB
        Xuan Gong
      4. YARN-3347.3.patch
        31 kB
        Xuan Gong
      5. YARN-3347.2.rebase.patch
        29 kB
        Xuan Gong
      6. YARN-3347.2.patch
        30 kB
        Xuan Gong
      7. YARN-3347.1.rebase.patch
        29 kB
        Xuan Gong
      8. YARN-3347.1.patch
        29 kB
        Xuan Gong

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2113 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2113/)
          YARN-3347. Improve YARN log command to get AMContainer logs as well as running containers logs. Contributed by Xuan Gong. (junping_du: rev a77d628339afaf2f5a085c73fd81a805b18348c9)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2113 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2113/ ) YARN-3347 . Improve YARN log command to get AMContainer logs as well as running containers logs. Contributed by Xuan Gong. (junping_du: rev a77d628339afaf2f5a085c73fd81a805b18348c9) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #164 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/164/)
          YARN-3347. Improve YARN log command to get AMContainer logs as well as running containers logs. Contributed by Xuan Gong. (junping_du: rev a77d628339afaf2f5a085c73fd81a805b18348c9)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #164 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/164/ ) YARN-3347 . Improve YARN log command to get AMContainer logs as well as running containers logs. Contributed by Xuan Gong. (junping_du: rev a77d628339afaf2f5a085c73fd81a805b18348c9) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #154 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/154/)
          YARN-3347. Improve YARN log command to get AMContainer logs as well as running containers logs. Contributed by Xuan Gong. (junping_du: rev a77d628339afaf2f5a085c73fd81a805b18348c9)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #154 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/154/ ) YARN-3347 . Improve YARN log command to get AMContainer logs as well as running containers logs. Contributed by Xuan Gong. (junping_du: rev a77d628339afaf2f5a085c73fd81a805b18348c9) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #897 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/897/)
          YARN-3347. Improve YARN log command to get AMContainer logs as well as running containers logs. Contributed by Xuan Gong. (junping_du: rev a77d628339afaf2f5a085c73fd81a805b18348c9)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #897 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/897/ ) YARN-3347 . Improve YARN log command to get AMContainer logs as well as running containers logs. Contributed by Xuan Gong. (junping_du: rev a77d628339afaf2f5a085c73fd81a805b18348c9) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2095 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2095/)
          YARN-3347. Improve YARN log command to get AMContainer logs as well as running containers logs. Contributed by Xuan Gong. (junping_du: rev a77d628339afaf2f5a085c73fd81a805b18348c9)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2095 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2095/ ) YARN-3347 . Improve YARN log command to get AMContainer logs as well as running containers logs. Contributed by Xuan Gong. (junping_du: rev a77d628339afaf2f5a085c73fd81a805b18348c9) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #163 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/163/)
          YARN-3347. Improve YARN log command to get AMContainer logs as well as running containers logs. Contributed by Xuan Gong. (junping_du: rev a77d628339afaf2f5a085c73fd81a805b18348c9)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #163 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/163/ ) YARN-3347 . Improve YARN log command to get AMContainer logs as well as running containers logs. Contributed by Xuan Gong. (junping_du: rev a77d628339afaf2f5a085c73fd81a805b18348c9) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7578 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7578/)
          YARN-3347. Improve YARN log command to get AMContainer logs as well as running containers logs. Contributed by Xuan Gong. (junping_du: rev a77d628339afaf2f5a085c73fd81a805b18348c9)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7578 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7578/ ) YARN-3347 . Improve YARN log command to get AMContainer logs as well as running containers logs. Contributed by Xuan Gong. (junping_du: rev a77d628339afaf2f5a085c73fd81a805b18348c9) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java hadoop-yarn-project/CHANGES.txt
          Hide
          djp Junping Du added a comment -

          I have commit v4 patch to trunk and branch-2. Thanks Xuan Gong for contributing the patch!

          Show
          djp Junping Du added a comment - I have commit v4 patch to trunk and branch-2. Thanks Xuan Gong for contributing the patch!
          Hide
          djp Junping Du added a comment -

          Latest patch LGTM. Verified locally that mvn eclipse:eclipse can pass too.
          +1. Will commit it shortly.

          Show
          djp Junping Du added a comment - Latest patch LGTM. Verified locally that mvn eclipse:eclipse can pass too. +1. Will commit it shortly.
          Hide
          xgong Xuan Gong added a comment -

          -1 eclipse:eclipse is not related. Can compile mvn eclipse:eclipse successfully locally

          Show
          xgong Xuan Gong added a comment - -1 eclipse:eclipse is not related. Can compile mvn eclipse:eclipse successfully locally
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12725015/YARN-3347.4.patch
          against trunk revision 7fc50e2.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          -1 eclipse:eclipse. The patch failed to build with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7319//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7319//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12725015/YARN-3347.4.patch against trunk revision 7fc50e2. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. -1 eclipse:eclipse . The patch failed to build with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7319//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7319//console This message is automatically generated.
          Hide
          xgong Xuan Gong added a comment -

          new patch addressed all the latest comments

          Show
          xgong Xuan Gong added a comment - new patch addressed all the latest comments
          Hide
          djp Junping Du added a comment -

          Thanks Xuan Gong for updating the patch! The patch looks good in overall. However, I think we need some unit test to cover some core functional methods, e.g readContainerLogsForALogType(), etc.
          In addition, a question on following code:

          +    if (amContainers.contains("ALL")) {
          +      for (AMLogsRequest request : requests) {
          +        outputAMContainerLogs(request, conf, appId, logFiles, logCliHelper,
          +          appOwner);
          +      }
          +    }
          +
          +    for (String amContainer : amContainers) {
          +      if (amContainer.equalsIgnoreCase("ALL")) {
          +        continue;
          +      }
          +      int amContainerId = Integer.parseInt(amContainer.trim());
          +      if (amContainerId == -1) {
          +        outputAMContainerLogs(requests.get(requests.size() - 1), conf, appId,
          +          logFiles, logCliHelper, appOwner);
          +      } else {
          +        if (amContainerId <= requests.size()) {
          +          outputAMContainerLogs(requests.get(amContainerId - 1), conf, appId,
          +            logFiles, logCliHelper, appOwner);
          +        }
          +      }
          +    }
          

          My question is: if user specify "ALL" and other attempts in command line, do we need to print out other AM attempts log after print all attempts log? From the code, it looks to do so but the behavior sounds a little weird.

          Show
          djp Junping Du added a comment - Thanks Xuan Gong for updating the patch! The patch looks good in overall. However, I think we need some unit test to cover some core functional methods, e.g readContainerLogsForALogType(), etc. In addition, a question on following code: + if (amContainers.contains( "ALL" )) { + for (AMLogsRequest request : requests) { + outputAMContainerLogs(request, conf, appId, logFiles, logCliHelper, + appOwner); + } + } + + for ( String amContainer : amContainers) { + if (amContainer.equalsIgnoreCase( "ALL" )) { + continue ; + } + int amContainerId = Integer .parseInt(amContainer.trim()); + if (amContainerId == -1) { + outputAMContainerLogs(requests.get(requests.size() - 1), conf, appId, + logFiles, logCliHelper, appOwner); + } else { + if (amContainerId <= requests.size()) { + outputAMContainerLogs(requests.get(amContainerId - 1), conf, appId, + logFiles, logCliHelper, appOwner); + } + } + } My question is: if user specify "ALL" and other attempts in command line, do we need to print out other AM attempts log after print all attempts log? From the code, it looks to do so but the behavior sounds a little weird.
          Hide
          xgong Xuan Gong added a comment -

          -1 CORE TESTS is not related

          Show
          xgong Xuan Gong added a comment - -1 CORE TESTS is not related
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12724216/YARN-3347.3.1.patch
          against trunk revision 6495940.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

          org.apache.hadoop.yarn.client.api.impl.TestAMRMClient

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7272//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7272//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724216/YARN-3347.3.1.patch against trunk revision 6495940. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.yarn.client.api.impl.TestAMRMClient Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7272//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7272//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12724191/YARN-3347.3.rebase.patch
          against trunk revision 6495940.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7269//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7269//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7269//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724191/YARN-3347.3.rebase.patch against trunk revision 6495940. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7269//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7269//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7269//console This message is automatically generated.
          Hide
          xgong Xuan Gong added a comment -

          Thanks for the review.

          For comma-separated value, do we mean attempt number? If so, may be we should describe more explicitly here? Also, can we use 0 (instead of -1) for AM container of latest attempt. If so, all negative value here is illegal.

          Added. I prefer to use -1 for the latest AM Container. 0 in the list/array is the first element.

          Missing ! before getConf()?

          Fixed

          Sounds like we are re-order the sequence of user's input which seems unnecessary to me. I would suggest to keep order from user's input or it could confuse people.

          Fixed

          Also, for method of run(String[] args), it looks very complexity for now. Can we do some refactor work there and put some comments inline?

          Yes, it indeed added some logics. Added some comments.

          Show
          xgong Xuan Gong added a comment - Thanks for the review. For comma-separated value, do we mean attempt number? If so, may be we should describe more explicitly here? Also, can we use 0 (instead of -1) for AM container of latest attempt. If so, all negative value here is illegal. Added. I prefer to use -1 for the latest AM Container. 0 in the list/array is the first element. Missing ! before getConf()? Fixed Sounds like we are re-order the sequence of user's input which seems unnecessary to me. I would suggest to keep order from user's input or it could confuse people. Fixed Also, for method of run(String[] args), it looks very complexity for now. Can we do some refactor work there and put some comments inline? Yes, it indeed added some logics. Added some comments.
          Hide
          djp Junping Du added a comment -

          Hi Xuan Gong, thanks for reporting this issue and delivering a patch for fixing this!
          This looks like a very helpful feature for trouble shooting. I went through the patch quickly and have some comments so far:

          +    Option amOption = new Option(AM_CONTAINER_OPTION, true, 
          +      "Prints the AM Container logs for this application. "
          +      + "Specify comma-separated value to get logs for related AM Container. "
          +      + "To get logs for all AM Containers, use -am ALL. "
          +      + "To get logs for the latest AM Container, use -am -1. "
          +      + "By default, it will only print out syslog. Work with -logFiles "
          +      + "to get other logs");
          

          For comma-separated value, do we mean attempt number? If so, may be we should describe more explicitly here? Also, can we use 0 (instead of -1) for AM container of latest attempt. If so, all negative value here is illegal.

          +        if (getConf().getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
          +          YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
          +          System.out.println("Please enable the application history service. Or ");
          +        }
          

          Missing ! before getConf()?

          In method of printAMContainerLogsForRunningApplication(),

          +    boolean printAll = amContainers.contains("ALL");
          +
          +    for (int i = 0; i < amContainersInfo.length(); i++) {
          +      boolean printThis = amContainers.contains(Integer.toString(i+1)) 
          +          || (i == (amContainersInfo.length()-1)
          +              && amContainers.contains(Integer.toString(-1)));
          +      if (printAll || printThis) {
          +        String nodeHttpAddress =
          +            amContainersInfo.getJSONObject(i).getString("nodeHttpAddress");
          +        String containerId =
          +            amContainersInfo.getJSONObject(i).getString("containerId");
          +        String nodeId = amContainersInfo.getJSONObject(i).getString("nodeId");
          +        if (nodeHttpAddress != null && containerId != null
          +            && !nodeHttpAddress.isEmpty() && !containerId.isEmpty()) {
          +          printContainerLogsFromRunningApplication(conf, appId, containerId,
          +            nodeHttpAddress, nodeId, logFiles, logCliHelper, appOwner);
          +        }
          +      }
          +    }
          +    return 0;
          +  }
          

          Sounds like we are re-order the sequence of user's input which seems unnecessary to me. I would suggest to keep order from user's input or it could confuse people. Also, the logic here sounds not quite straightforward. I would expect something simper, like pseudo code below:

          if (printAll) {
            go through amContainersInfo and print
          }
          for (amContainer : amContainers) {
            amContainer == -1? print amContainersInfo(last-one) : 
            print amContainersInfo(amContainer -1);
          }
          

          Also, for method of run(String[] args), it looks very complexity for now. Can we do some refactor work there and put some comments inline?

          Show
          djp Junping Du added a comment - Hi Xuan Gong , thanks for reporting this issue and delivering a patch for fixing this! This looks like a very helpful feature for trouble shooting. I went through the patch quickly and have some comments so far: + Option amOption = new Option(AM_CONTAINER_OPTION, true , + "Prints the AM Container logs for this application. " + + "Specify comma-separated value to get logs for related AM Container. " + + "To get logs for all AM Containers, use -am ALL. " + + "To get logs for the latest AM Container, use -am -1. " + + "By default , it will only print out syslog. Work with -logFiles " + + "to get other logs" ); For comma-separated value, do we mean attempt number? If so, may be we should describe more explicitly here? Also, can we use 0 (instead of -1) for AM container of latest attempt. If so, all negative value here is illegal. + if (getConf().getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED, + YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) { + System .out.println( "Please enable the application history service. Or " ); + } Missing ! before getConf()? In method of printAMContainerLogsForRunningApplication(), + boolean printAll = amContainers.contains( "ALL" ); + + for ( int i = 0; i < amContainersInfo.length(); i++) { + boolean printThis = amContainers.contains( Integer .toString(i+1)) + || (i == (amContainersInfo.length()-1) + && amContainers.contains( Integer .toString(-1))); + if (printAll || printThis) { + String nodeHttpAddress = + amContainersInfo.getJSONObject(i).getString( "nodeHttpAddress" ); + String containerId = + amContainersInfo.getJSONObject(i).getString( "containerId" ); + String nodeId = amContainersInfo.getJSONObject(i).getString( "nodeId" ); + if (nodeHttpAddress != null && containerId != null + && !nodeHttpAddress.isEmpty() && !containerId.isEmpty()) { + printContainerLogsFromRunningApplication(conf, appId, containerId, + nodeHttpAddress, nodeId, logFiles, logCliHelper, appOwner); + } + } + } + return 0; + } Sounds like we are re-order the sequence of user's input which seems unnecessary to me. I would suggest to keep order from user's input or it could confuse people. Also, the logic here sounds not quite straightforward. I would expect something simper, like pseudo code below: if (printAll) { go through amContainersInfo and print } for (amContainer : amContainers) { amContainer == -1? print amContainersInfo(last-one) : print amContainersInfo(amContainer -1); } Also, for method of run(String[] args), it looks very complexity for now. Can we do some refactor work there and put some comments inline?
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12706818/YARN-3347.2.rebase.patch
          against trunk revision 9fae455.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7087//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7087//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12706818/YARN-3347.2.rebase.patch against trunk revision 9fae455. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7087//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7087//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12706770/YARN-3347.2.patch
          against trunk revision 2c238ae.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7083//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7083//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12706770/YARN-3347.2.patch against trunk revision 2c238ae. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7083//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7083//console This message is automatically generated.
          Hide
          xgong Xuan Gong added a comment -

          fix -1 on findBug

          Show
          xgong Xuan Gong added a comment - fix -1 on findBug
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12706154/YARN-3347.1.rebase.patch
          against trunk revision e1feb4e.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 5 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7063//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7063//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7063//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12706154/YARN-3347.1.rebase.patch against trunk revision e1feb4e. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 5 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7063//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7063//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7063//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12705973/YARN-3347.1.patch
          against trunk revision 1561231.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
          org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService

          The following test timeouts occurred in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.TestRM

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7045//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7045//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12705973/YARN-3347.1.patch against trunk revision 1561231. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 6 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService The following test timeouts occurred in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestRM Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7045//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7045//console This message is automatically generated.
          Hide
          xgong Xuan Gong added a comment -

          In the same patch, we also show the logs for the running containers.

          Show
          xgong Xuan Gong added a comment - In the same patch, we also show the logs for the running containers.

            People

            • Assignee:
              xgong Xuan Gong
              Reporter:
              xgong Xuan Gong
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development