Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5743

[Atsv2] Publish queue name and RMAppMetrics to ATS

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      App queue name is missed to publish to ATSv2.
      And RMAppMetrcs publish only cpu and memory. There are many more things to publish from app metrics such as
      resourcePreempted;
      numNonAMContainersPreempted;
      numAMContainersPreempted.

      And RMAppMetrics need to be published to App metrics rather than info.

      1. 0001-YARN-5743.patch
        10 kB
        Rohith Sharma K S
      2. 0002-YARN-5743.patch
        10 kB
        Rohith Sharma K S

        Activity

        Hide
        varun_saxena Varun Saxena added a comment -

        Thanks Rohith Sharma K S for filing the JIRA. All of this makes sense. Capturing this information as metrics will help in this being aggregated upto flow run and flow level as well.
        Also, we can make sure that the information which we serve from RM/NM REST endpoints currently can be published and hence served from ATS too (if the information makes sense as historical data).

        Show
        varun_saxena Varun Saxena added a comment - Thanks Rohith Sharma K S for filing the JIRA. All of this makes sense. Capturing this information as metrics will help in this being aggregated upto flow run and flow level as well. Also, we can make sure that the information which we serve from RM/NM REST endpoints currently can be published and hence served from ATS too (if the information makes sense as historical data).
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Update the patch for publishing queue name and RMAppMetrics

        Show
        rohithsharma Rohith Sharma K S added a comment - Update the patch for publishing queue name and RMAppMetrics
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 12s Maven dependency ordering for branch
        +1 mvninstall 8m 20s trunk passed
        +1 compile 1m 39s trunk passed
        +1 checkstyle 0m 31s trunk passed
        +1 mvnsite 0m 59s trunk passed
        +1 mvneclipse 0m 29s trunk passed
        +1 findbugs 1m 36s trunk passed
        +1 javadoc 0m 37s trunk passed
        0 mvndep 0m 8s Maven dependency ordering for patch
        +1 mvninstall 0m 50s the patch passed
        +1 compile 1m 32s the patch passed
        +1 javac 1m 32s the patch passed
        +1 checkstyle 0m 30s the patch passed
        +1 mvnsite 0m 56s the patch passed
        +1 mvneclipse 0m 25s the patch passed
        -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 findbugs 1m 46s the patch passed
        +1 javadoc 0m 33s the patch passed
        +1 unit 0m 26s hadoop-yarn-server-common in the patch passed.
        +1 unit 45m 23s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        68m 13s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833730/0001-YARN-5743.patch
        JIRA Issue YARN-5743
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 596e3d8fe53a 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ed9fcbe
        Default Java 1.8.0_101
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13407/artifact/patchprocess/whitespace-eol.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13407/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13407/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 8m 20s trunk passed +1 compile 1m 39s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 1m 36s trunk passed +1 javadoc 0m 37s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 50s the patch passed +1 compile 1m 32s the patch passed +1 javac 1m 32s the patch passed +1 checkstyle 0m 30s the patch passed +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 25s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 33s the patch passed +1 unit 0m 26s hadoop-yarn-server-common in the patch passed. +1 unit 45m 23s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 68m 13s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833730/0001-YARN-5743.patch JIRA Issue YARN-5743 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 596e3d8fe53a 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ed9fcbe Default Java 1.8.0_101 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13407/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13407/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Console output https://builds.apache.org/job/PreCommit-YARN-Build/13407/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        varun_saxena Varun Saxena added a comment -

        Thanks Rohith Sharma K S for the patch.
        Overall the patch looks good.

        1. Suffixing each metric with _METRIC is not required as we are storing them as metric, so its self evident. Its just few extra unnecessary bytes being stored.
        2. Cosmetic comment - Maybe we can make the formatting of statements inside getTimelinelineAppMetrics consistent.
        Show
        varun_saxena Varun Saxena added a comment - Thanks Rohith Sharma K S for the patch. Overall the patch looks good. Suffixing each metric with _METRIC is not required as we are storing them as metric, so its self evident. Its just few extra unnecessary bytes being stored. Cosmetic comment - Maybe we can make the formatting of statements inside getTimelinelineAppMetrics consistent.
        Hide
        varun_saxena Varun Saxena added a comment -

        By the way we should probably publish log aggregation status and diagnostics too. This will help, if user cannot find the application logs in designated location due to some failure during aggregation.

        Show
        varun_saxena Varun Saxena added a comment - By the way we should probably publish log aggregation status and diagnostics too. This will help, if user cannot find the application logs in designated location due to some failure during aggregation.
        Hide
        sjlee0 Sangjin Lee added a comment -

        Thanks for filing this issue and the patch Rohith Sharma K S. It looks quite reasonable. Please address Varun's comments and we should be close.

        Do we want this on the YARN-5355 branch only or on the trunk as well?

        Show
        sjlee0 Sangjin Lee added a comment - Thanks for filing this issue and the patch Rohith Sharma K S . It looks quite reasonable. Please address Varun's comments and we should be close. Do we want this on the YARN-5355 branch only or on the trunk as well?
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        By the way we should probably publish log aggregation status and diagnostics too. This will help, if user cannot find the application logs in designated location due to some failure during aggregation.

        I think we should NOT store log aggregation status and its diagnostic information because of couple of reasons

        1. aggregated log will be deleted after some time as per configuration configured. Since we are storing very long history of applications, very old applications logs might be deleted which gives wrong information to reader.
        2. Diagnostic report is for per node basis. This will over spam the application storage. Let say, if aggregation fails 100 nodes may be because of ticket, then full of application entity will be filled with list of nodeId vs diagnostic information.
        Show
        rohithsharma Rohith Sharma K S added a comment - By the way we should probably publish log aggregation status and diagnostics too. This will help, if user cannot find the application logs in designated location due to some failure during aggregation. I think we should NOT store log aggregation status and its diagnostic information because of couple of reasons aggregated log will be deleted after some time as per configuration configured. Since we are storing very long history of applications, very old applications logs might be deleted which gives wrong information to reader. Diagnostic report is for per node basis. This will over spam the application storage. Let say, if aggregation fails 100 nodes may be because of ticket, then full of application entity will be filled with list of nodeId vs diagnostic information.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Updated patch addressing other comments..

        Show
        rohithsharma Rohith Sharma K S added a comment - Updated patch addressing other comments..
        Hide
        varun_saxena Varun Saxena added a comment -

        Diagnostic report is for per node basis. This will over spam the application storage.

        That's true. Agree with you.

        aggregated log will be deleted after some time as per configuration configured. Since we are storing very long history of applications, very old applications logs might be deleted which gives wrong information to reader.

        I do not think it will give wrong information to reader because its just saying whether log aggregation had succeeded or failed. Deletion of aggregated logs is a separate functionality which admins would be knowing about.
        I just thought about this because for some applications which may have failed in last 3-4 days, we may look for their logs and if some logs were missing, log aggregation status and diagnostics info may have been useful. But without diagnostic info, we may not get much info on what went wrong anyways. Also, log aggregation status will not be immediately available i.e. when finished event is published. So even if we do publish it, we can probably handle it in a different JIRA.
        Sangjin Lee, your thoughts on this ?

        Show
        varun_saxena Varun Saxena added a comment - Diagnostic report is for per node basis. This will over spam the application storage. That's true. Agree with you. aggregated log will be deleted after some time as per configuration configured. Since we are storing very long history of applications, very old applications logs might be deleted which gives wrong information to reader. I do not think it will give wrong information to reader because its just saying whether log aggregation had succeeded or failed. Deletion of aggregated logs is a separate functionality which admins would be knowing about. I just thought about this because for some applications which may have failed in last 3-4 days, we may look for their logs and if some logs were missing, log aggregation status and diagnostics info may have been useful. But without diagnostic info, we may not get much info on what went wrong anyways. Also, log aggregation status will not be immediately available i.e. when finished event is published. So even if we do publish it, we can probably handle it in a different JIRA. Sangjin Lee , your thoughts on this ?
        Hide
        varun_saxena Varun Saxena added a comment -

        Overall I am +1 on the latest patch pending Jenkins...
        Will commit it later today unless there are further comments. This I assume needs to go in trunk. YARN-5355 branch, it can come to, when we do trunk rebase.

        Show
        varun_saxena Varun Saxena added a comment - Overall I am +1 on the latest patch pending Jenkins... Will commit it later today unless there are further comments. This I assume needs to go in trunk. YARN-5355 branch, it can come to, when we do trunk rebase.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Yes, this need to get committed in trunk.

        Show
        rohithsharma Rohith Sharma K S added a comment - Yes, this need to get committed in trunk.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 53s Maven dependency ordering for branch
        +1 mvninstall 6m 54s trunk passed
        +1 compile 1m 31s trunk passed
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 0m 58s trunk passed
        +1 mvneclipse 0m 29s trunk passed
        +1 findbugs 1m 32s trunk passed
        +1 javadoc 0m 34s trunk passed
        0 mvndep 0m 8s Maven dependency ordering for patch
        +1 mvninstall 0m 45s the patch passed
        +1 compile 1m 25s the patch passed
        +1 javac 1m 25s the patch passed
        +1 checkstyle 0m 26s the patch passed
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 24s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 41s the patch passed
        +1 javadoc 0m 29s the patch passed
        +1 unit 0m 24s hadoop-yarn-server-common in the patch passed.
        +1 unit 34m 40s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        55m 52s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833975/0002-YARN-5743.patch
        JIRA Issue YARN-5743
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8466f0d37794 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d26a1bb
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13420/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13420/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 53s Maven dependency ordering for branch +1 mvninstall 6m 54s trunk passed +1 compile 1m 31s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 1m 32s trunk passed +1 javadoc 0m 34s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 0m 45s the patch passed +1 compile 1m 25s the patch passed +1 javac 1m 25s the patch passed +1 checkstyle 0m 26s the patch passed +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 41s the patch passed +1 javadoc 0m 29s the patch passed +1 unit 0m 24s hadoop-yarn-server-common in the patch passed. +1 unit 34m 40s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 55m 52s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833975/0002-YARN-5743.patch JIRA Issue YARN-5743 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8466f0d37794 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d26a1bb Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13420/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Console output https://builds.apache.org/job/PreCommit-YARN-Build/13420/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        sjlee0 Sangjin Lee added a comment -

        +1. Varun Saxena, please don't forget to cherry-pick into the YARN-5355-branch-2 branch.

        Show
        sjlee0 Sangjin Lee added a comment - +1. Varun Saxena , please don't forget to cherry-pick into the YARN-5355 -branch-2 branch.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10631 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10631/)
        YARN-5743. [Atsv2] Publish queue name and RMAppMetrics to ATS (Rohith (varunsaxena: rev b154d3edcee95254d41c237142870f39e826a519)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ApplicationMetricsConstants.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisherForV2.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TimelineServiceV2Publisher.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10631 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10631/ ) YARN-5743 . [Atsv2] Publish queue name and RMAppMetrics to ATS (Rohith (varunsaxena: rev b154d3edcee95254d41c237142870f39e826a519) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ApplicationMetricsConstants.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisherForV2.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TimelineServiceV2Publisher.java
        Hide
        varun_saxena Varun Saxena added a comment -

        Committed to trunk, YARN-5355 and YARN-5355-branch-2.
        Thanks Rohith Sharma K S for your contribution and Sangjin Lee for reviews.

        Show
        varun_saxena Varun Saxena added a comment - Committed to trunk, YARN-5355 and YARN-5355 -branch-2. Thanks Rohith Sharma K S for your contribution and Sangjin Lee for reviews.

          People

          • Assignee:
            rohithsharma Rohith Sharma K S
            Reporter:
            rohithsharma Rohith Sharma K S
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development