Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10353

Log vcores used and cumulative cpu in containers monitor

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.4.0
    • 3.2.2, 3.4.0, 3.3.1
    • yarn
    • None

    Description

      We currently log the percentage/cpu and percentage/cpus-used-by-yarn in the Containers Monitor log. It would be useful to also log vcores used vs vcores assigned, and total accumulated CPU time.

      For example, currently we have an audit log that looks like this:

      2020-07-16 20:33:51,550 DEBUG [Container Monitor] ContainersMonitorImpl.audit (ContainersMonitorImpl.java:recordUsage(651)) - Resource usage of ProcessTree 809 for container-id container_1594931466123_0002_01_000007: 309.5 MB of 2 GB physical memory used; 2.8 GB of 4.2 GB virtual memory used CPU:143.0905 CPU/core:35.772625
      

      The proposal is to add two more fields to show vCores and Cumulative CPU ms:

      2020-07-16 20:33:51,550 DEBUG [Container Monitor] ContainersMonitorImpl.audit (ContainersMonitorImpl.java:recordUsage(651)) - Resource usage of ProcessTree 809 for container-id container_1594931466123_0002_01_000007: 309.5 MB of 2 GB physical memory used; 2.8 GB of 4.2 GB virtual memory used CPU:143.0905 CPU/core:35.772625 vCores:2/1 CPU-ms:4180
      

      This is a snippet of a log from one of our clusters running branch-2.8 with a similar change.

      2020-07-16 21:00:02,240 [Container Monitor] DEBUG ContainersMonitorImpl.audit: Memory usage of ProcessTree 5267 for container-id container_e04_1594079801456_1397450_01_001992: 1.6 GB of 2.5 GB physical memory used; 3.8 GB of 5.3 GB virtual memory used. CPU usage: 18 of 10 CPU vCores used. Cumulative CPU time: 157410
      2020-07-16 21:00:02,269 [Container Monitor] DEBUG ContainersMonitorImpl.audit: Memory usage of ProcessTree 18801 for container-id container_e04_1594079801456_1390375_01_000019: 413.2 MB of 2.5 GB physical memory used; 3.8 GB of 5.3 GB virtual memory used. CPU usage: 0 of 10 CPU vCores used. Cumulative CPU time: 113830
      2020-07-16 21:00:02,298 [Container Monitor] DEBUG ContainersMonitorImpl.audit: Memory usage of ProcessTree 5279 for container-id container_e04_1594079801456_1397450_01_001991: 2.2 GB of 2.5 GB physical memory used; 3.8 GB of 5.3 GB virtual memory used. CPU usage: 17 of 10 CPU vCores used. Cumulative CPU time: 128630
      2020-07-16 21:00:02,339 [Container Monitor] DEBUG ContainersMonitorImpl.audit: Memory usage of ProcessTree 24189 for container-id container_e04_1594079801456_1390430_01_000415: 392.7 MB of 2.5 GB physical memory used; 3.8 GB of 5.3 GB virtual memory used. CPU usage: 0 of 10 CPU vCores used. Cumulative CPU time: 96060
      2020-07-16 21:00:02,367 [Container Monitor] DEBUG ContainersMonitorImpl.audit: Memory usage of ProcessTree 6751 for container-id container_e04_1594079801456_1397923_01_003248: 1.3 GB of 3 GB physical memory used; 4.3 GB of 6.3 GB virtual memory used. CPU usage: 12 of 10 CPU vCores used. Cumulative CPU time: 116820
      2020-07-16 21:00:02,396 [Container Monitor] DEBUG ContainersMonitorImpl.audit: Memory usage of ProcessTree 12138 for container-id container_e04_1594079801456_1397760_01_000044: 4.4 GB of 6 GB physical memory used; 6.9 GB of 12.6 GB virtual memory used. CPU usage: 15 of 10 CPU vCores used. Cumulative CPU time: 45900
      2020-07-16 21:00:02,424 [Container Monitor] DEBUG ContainersMonitorImpl.audit: Memory usage of ProcessTree 101918 for container-id container_e04_1594079801456_1391130_01_002378: 2.4 GB of 4 GB physical memory used; 5.8 GB of 8.4 GB virtual memory used. CPU usage: 13 of 10 CPU vCores used. Cumulative CPU time: 2572390
      2020-07-16 21:00:02,456 [Container Monitor] DEBUG ContainersMonitorImpl.audit: Memory usage of ProcessTree 26596 for container-id container_e04_1594079801456_1390446_01_000665: 418.6 MB of 2.5 GB physical memory used; 3.8 GB of 5.3 GB virtual memory used. CPU usage: 0 of 10 CPU vCores used. Cumulative CPU time: 101210
      

      Attachments

        1. YARN-10353-branch-3.2.002.patch
          2 kB
          Eric Payne
        2. YARN-10353.002.patch
          2 kB
          Jim Brennan
        3. YARN-10353.001.patch
          2 kB
          Jim Brennan

        Activity

          People

            jbrennan Jim Brennan
            jbrennan Jim Brennan
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: