Details

    • Hadoop Flags:
      Reviewed

      Description

      YARN-3022 exposed container metrics from node manager, but document is missing in Metrics.md.

      1. YARN-6028.01.patch
        6 kB
        Weiwei Yang
      2. YARN-6028.02.patch
        6 kB
        Weiwei Yang
      3. YARN-6028.03.patch
        6 kB
        Weiwei Yang
      4. YARN-6028.04.patch
        6 kB
        Weiwei Yang

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvninstall 13m 10s trunk passed
        +1 mvnsite 1m 3s trunk passed
        +1 mvnsite 0m 58s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        16m 0s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6028
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845025/YARN-6028.01.patch
        Optional Tests asflicense mvnsite
        uname Linux cdce3730ef9e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / a4f6665
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14490/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 13m 10s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvnsite 0m 58s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 16m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6028 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845025/YARN-6028.01.patch Optional Tests asflicense mvnsite uname Linux cdce3730ef9e 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a4f6665 modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/14490/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        templedf Daniel Templeton added a comment -

        Thanks, Weiwei Yang! Much needed doc. A few comments:

        • Milliseconds is "ms". "MS" is mega-Siemens
        • "vcore usages" should be "vcores used" throughout
        • "Total number of physical memory" should be "Total amount of physical memory" throughout
        • I think "usage[s]" can be globally replaced with "used". A "usage" is a single event, so if I used 16 vcores once, that's 1 usage with 16 vcores used.
        Show
        templedf Daniel Templeton added a comment - Thanks, Weiwei Yang ! Much needed doc. A few comments: Milliseconds is "ms". "MS" is mega-Siemens "vcore usages" should be "vcores used" throughout "Total number of physical memory" should be "Total amount of physical memory" throughout I think "usage [s] " can be globally replaced with "used". A "usage" is a single event, so if I used 16 vcores once, that's 1 usage with 16 vcores used .
        Hide
        cheersyang Weiwei Yang added a comment -

        Hello Daniel Templeton

        Thanks for your comments, I have made following modifications based on your suggestions,

        1. MS to msec
        2. vcore usages to vcore used
        3. Revised "Total number of ..." to "Total number of ... metrics", I think these metrics, such as PMemUsageMBsNumUsage, PMemUsageMBsINumUsage, PCpuUsagePercentNumUsage, PCpuUsagePercentINumUsage, they were describing the total number of samples since beginning or in an interval
        4. Globally replaced usage(s) with used

        Hope it makes sense
        Thanks

        Show
        cheersyang Weiwei Yang added a comment - Hello Daniel Templeton Thanks for your comments, I have made following modifications based on your suggestions, MS to msec vcore usages to vcore used Revised "Total number of ..." to "Total number of ... metrics" , I think these metrics, such as PMemUsageMBsNumUsage , PMemUsageMBsINumUsage , PCpuUsagePercentNumUsage , PCpuUsagePercentINumUsage , they were describing the total number of samples since beginning or in an interval Globally replaced usage(s) with used Hope it makes sense Thanks
        Hide
        cheersyang Weiwei Yang added a comment -

        For some reason, the jenkins job did not get triggered automatically when I attach new patches...

        Show
        cheersyang Weiwei Yang added a comment - For some reason, the jenkins job did not get triggered automatically when I attach new patches...
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvninstall 14m 27s trunk passed
        +1 mvnsite 1m 4s trunk passed
        +1 mvnsite 0m 58s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        17m 21s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6028
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845463/YARN-6028.02.patch
        Optional Tests asflicense mvnsite
        uname Linux 807a0b17803f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8fadd69
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14542/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 14m 27s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvnsite 0m 58s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 17m 21s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6028 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845463/YARN-6028.02.patch Optional Tests asflicense mvnsite uname Linux 807a0b17803f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8fadd69 modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/14542/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        cheersyang Weiwei Yang added a comment -

        Resubmitted the patch, now jenkins got happy. Hi Daniel Templeton, how about the v2 patch looks? Does my previous comment here make sense to you? Please let me know, thank you.

        Show
        cheersyang Weiwei Yang added a comment - Resubmitted the patch, now jenkins got happy. Hi Daniel Templeton , how about the v2 patch looks? Does my previous comment here make sense to you? Please let me know, thank you.
        Hide
        templedf Daniel Templeton added a comment -

        Thanks, Weiwei Yang. The changes look good. In reviewing the new patch, though, I found a couple more issues:

        • "Minimal" should probably be "minimum," e.g. "minimal physical CPU core percent used"
        • "1000 times of" should be "1000 times the," e.g. "1000 times the minimum vcores used"
        • "vcore used" should be "vcores used," e.g. "1000 times the minimum vcores used"
        Show
        templedf Daniel Templeton added a comment - Thanks, Weiwei Yang . The changes look good. In reviewing the new patch, though, I found a couple more issues: "Minimal" should probably be "minimum," e.g. "minimal physical CPU core percent used" "1000 times of" should be "1000 times the," e.g. "1000 times the minimum vcores used" "vcore used" should be "vcores used," e.g. "1000 times the minimum vcores used"
        Hide
        cheersyang Weiwei Yang added a comment -

        Hello Daniel Templeton

        Appreciate your help to review this, sorry I am not a native English speaker so it might give you more trouble
        I just updated the patch according to your comments, uploaded v3. Thanks!

        Show
        cheersyang Weiwei Yang added a comment - Hello Daniel Templeton Appreciate your help to review this, sorry I am not a native English speaker so it might give you more trouble I just updated the patch according to your comments, uploaded v3. Thanks!
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvninstall 17m 26s trunk passed
        +1 mvnsite 1m 8s trunk passed
        +1 mvnsite 1m 5s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        20m 35s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6028
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847966/YARN-6028.03.patch
        Optional Tests asflicense mvnsite
        uname Linux 8a2de65d7548 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e224c96
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14684/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 17m 26s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvnsite 1m 5s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 20m 35s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6028 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847966/YARN-6028.03.patch Optional Tests asflicense mvnsite uname Linux 8a2de65d7548 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e224c96 modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/14684/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        templedf Daniel Templeton added a comment -

        I'm just happy we're going to be getting complete docs! One more thing I should have said more explicitly before: the same vcore->vcores applies for core->cores.

        Show
        templedf Daniel Templeton added a comment - I'm just happy we're going to be getting complete docs! One more thing I should have said more explicitly before: the same vcore->vcores applies for core->cores.
        Hide
        cheersyang Weiwei Yang added a comment -

        Thanks Daniel Templeton, me too. Just uploaded v4 patch to use cores at all places.

        Show
        cheersyang Weiwei Yang added a comment - Thanks Daniel Templeton , me too. Just uploaded v4 patch to use cores at all places.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvninstall 14m 54s trunk passed
        +1 mvnsite 1m 2s trunk passed
        +1 mvnsite 0m 56s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        17m 44s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6028
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848199/YARN-6028.04.patch
        Optional Tests asflicense mvnsite
        uname Linux 44da71abc92d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 383aa9c
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14698/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 14m 54s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvnsite 0m 56s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 17m 44s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6028 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12848199/YARN-6028.04.patch Optional Tests asflicense mvnsite uname Linux 44da71abc92d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 383aa9c modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/14698/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        templedf Daniel Templeton added a comment -

        LGTM +1

        Show
        templedf Daniel Templeton added a comment - LGTM +1
        Hide
        templedf Daniel Templeton added a comment -

        Committed to trunk and branch-2. Thanks for the patch, Weiwei Yang!

        Show
        templedf Daniel Templeton added a comment - Committed to trunk and branch-2. Thanks for the patch, Weiwei Yang !
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11148 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11148/)
        YARN-6028. Add document for container metrics (Contributed by Weiwei (templedf: rev f5839fd6df77786353160b41eb4b5ceea9380017)

        • (edit) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11148 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11148/ ) YARN-6028 . Add document for container metrics (Contributed by Weiwei (templedf: rev f5839fd6df77786353160b41eb4b5ceea9380017) (edit) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
        Hide
        cheersyang Weiwei Yang added a comment -

        Thanks a lot Daniel Templeton

        Show
        cheersyang Weiwei Yang added a comment - Thanks a lot Daniel Templeton

          People

          • Assignee:
            cheersyang Weiwei Yang
            Reporter:
            cheersyang Weiwei Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development