Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      YARN should be aware of the resource utilization of the nodes when scheduling containers. For this, this task will implement the collection of memory/cpu usage on the node.

      1. YARN-3534-9.patch
        8 kB
        Íñigo Goiri
      2. YARN-3534-8.patch
        9 kB
        Íñigo Goiri
      3. YARN-3534-7.patch
        9 kB
        Íñigo Goiri
      4. YARN-3534-6.patch
        9 kB
        Íñigo Goiri
      5. YARN-3534-5.patch
        29 kB
        Íñigo Goiri
      6. YARN-3534-4.patch
        9 kB
        Íñigo Goiri
      7. YARN-3534-3.patch
        39 kB
        Íñigo Goiri
      8. YARN-3534-3.patch
        39 kB
        Íñigo Goiri
      9. YARN-3534-2.patch
        39 kB
        Íñigo Goiri
      10. YARN-3534-18.patch
        16 kB
        Íñigo Goiri
      11. YARN-3534-17.patch
        15 kB
        Íñigo Goiri
      12. YARN-3534-17.patch
        15 kB
        Íñigo Goiri
      13. YARN-3534-16.patch
        15 kB
        Íñigo Goiri
      14. YARN-3534-16.patch
        15 kB
        Íñigo Goiri
      15. YARN-3534-15.patch
        12 kB
        Íñigo Goiri
      16. YARN-3534-14.patch
        11 kB
        Íñigo Goiri
      17. YARN-3534-12.patch
        18 kB
        Íñigo Goiri
      18. YARN-3534-11.patch
        17 kB
        Íñigo Goiri
      19. YARN-3534-10.patch
        15 kB
        Íñigo Goiri
      20. YARN-3534-1.patch
        21 kB
        Íñigo Goiri

        Issue Links

          Activity

          Hide
          srikanthkandula Srikanth Kandula added a comment -

          Íñigo Goiri, Karthik Kambatla, could you comment on extending this to also take in network and disk usage information?

          Show
          srikanthkandula Srikanth Kandula added a comment - Íñigo Goiri , Karthik Kambatla , could you comment on extending this to also take in network and disk usage information?
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2236 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2236/)
          YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2236 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2236/ ) YARN-3534 . Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #279 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/279/)
          YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #279 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/279/ ) YARN-3534 . Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2217 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2217/)
          YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2217 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2217/ ) YARN-3534 . Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #290 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/290/)
          YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #290 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/290/ ) YARN-3534 . Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1020 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1020/)
          YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1020 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1020/ ) YARN-3534 . Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Hide
          elgoiri Íñigo Goiri added a comment -

          Thank you Karthik Kambatla for taking care of the review and the commits.
          I'll be moving to propagating this info to the scheduler.

          Show
          elgoiri Íñigo Goiri added a comment - Thank you Karthik Kambatla for taking care of the review and the commits. I'll be moving to propagating this info to the scheduler.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #286 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/286/)
          YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #286 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/286/ ) YARN-3534 . Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8311 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8311/)
          YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8311 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8311/ ) YARN-3534 . Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          Hide
          kasha Karthik Kambatla added a comment -

          Just committed to trunk and branch-2.

          Thanks Íñigo Goiri for the contribution and patience through the reviews.

          Show
          kasha Karthik Kambatla added a comment - Just committed to trunk and branch-2. Thanks Íñigo Goiri for the contribution and patience through the reviews.
          Hide
          kasha Karthik Kambatla added a comment -

          +1

          Show
          kasha Karthik Kambatla added a comment - +1
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 35s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 39s There were no new javac warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 46s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 23s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common.
          +1 yarn tests 6m 10s Tests passed in hadoop-yarn-server-nodemanager.
              53m 33s  



          Reason Tests
          Failed unit tests hadoop.yarn.util.TestRackResolver



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12750666/YARN-3534-18.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 8dfec7a
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8851/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8851/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8851/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8851/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8851/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8851/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 35s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 39s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 46s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 23s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common. +1 yarn tests 6m 10s Tests passed in hadoop-yarn-server-nodemanager.     53m 33s   Reason Tests Failed unit tests hadoop.yarn.util.TestRackResolver Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12750666/YARN-3534-18.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8dfec7a checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8851/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8851/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8851/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8851/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8851/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8851/console This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          I uploaded 17 twice instead of 18... sorry.

          Show
          elgoiri Íñigo Goiri added a comment - I uploaded 17 twice instead of 18... sorry.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Fixed xml defaults.

          Show
          elgoiri Íñigo Goiri added a comment - Fixed xml defaults.
          Hide
          kasha Karthik Kambatla added a comment -

          The latest patches (v17) don't seem to have the updates. Íñigo Goiri - could you please verify?

          Show
          kasha Karthik Kambatla added a comment - The latest patches (v17) don't seem to have the updates. Íñigo Goiri - could you please verify?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 52s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 45s There were no new javac warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 22s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 20s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common.
          +1 yarn tests 6m 8s Tests passed in hadoop-yarn-server-nodemanager.
              53m 54s  



          Reason Tests
          Failed unit tests hadoop.yarn.util.TestRackResolver



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12749435/YARN-3534-17.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 8f73bdd
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8802/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8802/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8802/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8802/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8802/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8802/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 52s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 45s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 22s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 20s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common. +1 yarn tests 6m 8s Tests passed in hadoop-yarn-server-nodemanager.     53m 54s   Reason Tests Failed unit tests hadoop.yarn.util.TestRackResolver Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749435/YARN-3534-17.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8f73bdd checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8802/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8802/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8802/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8802/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8802/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8802/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 45s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 41s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 45s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 3s The patch has no lines that end in whitespace.
          +1 install 1m 22s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          -1 yarn tests 1m 54s Tests failed in hadoop-yarn-common.
          -1 yarn tests 6m 7s Tests failed in hadoop-yarn-server-nodemanager.
              53m 35s  



          Reason Tests
          Failed unit tests hadoop.yarn.util.TestRackResolver
            hadoop.yarn.server.nodemanager.TestDeletionService



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12749432/YARN-3534-17.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 8f73bdd
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8801/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8801/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8801/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8801/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8801/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8801/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 45s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 45s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 22s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. -1 yarn tests 1m 54s Tests failed in hadoop-yarn-common. -1 yarn tests 6m 7s Tests failed in hadoop-yarn-server-nodemanager.     53m 35s   Reason Tests Failed unit tests hadoop.yarn.util.TestRackResolver   hadoop.yarn.server.nodemanager.TestDeletionService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749432/YARN-3534-17.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8f73bdd checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8801/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8801/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8801/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8801/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8801/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8801/console This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Fixed yarn-default.xml description.

          Show
          elgoiri Íñigo Goiri added a comment - Fixed yarn-default.xml description.
          Hide
          kasha Karthik Kambatla added a comment -

          In yarn-default.xml, for the container-mon configs, we should add info to say - "If not set, the value for nm-resource-monitor configs will be used."

          Show
          kasha Karthik Kambatla added a comment - In yarn-default.xml, for the container-mon configs, we should add info to say - "If not set, the value for nm-resource-monitor configs will be used."
          Hide
          elgoiri Íñigo Goiri added a comment -

          Incorporated Karthik Kambatla improvements but still not clear on the yarn-default.xml.

          Show
          elgoiri Íñigo Goiri added a comment - Incorporated Karthik Kambatla improvements but still not clear on the yarn-default.xml.
          Hide
          elgoiri Íñigo Goiri added a comment -
          1. I missed the Deprecated for the calculator... my bad. Fixed.
          2. That's good; fixed.
          3. The behavior for the CONTAINER_MON hasn't really changed, it's the node ones that are used a backups right? I fixed the default value and extended the resource monitor; not sure about the best wording though.
          Show
          elgoiri Íñigo Goiri added a comment - I missed the Deprecated for the calculator... my bad. Fixed. That's good; fixed. The behavior for the CONTAINER_MON hasn't really changed, it's the node ones that are used a backups right? I fixed the default value and extended the resource monitor; not sure about the best wording though.
          Hide
          kasha Karthik Kambatla added a comment -

          Hopefully, last set of comments:

          1. We should also not deprecate NM_CONTAINER_MON_RESOURCE_CALCULATOR. Should have been more explicit in my previous comment.
          2. ContainersMonitorImpl should prioritize container-specific configs and fallback if they are not present. Something along these lines:
                this.monitoringInterval =
                    conf.getLong(YarnConfiguration.NM_CONTAINER_MON_INTERVAL_MS,
                        conf.getLong(YarnConfiguration.NM_RESOURCE_MON_INTERVAL_MS,
                            YarnConfiguration.DEFAULT_NM_RESOURCE_MON_INTERVAL_MS));
            
                Class<? extends ResourceCalculatorPlugin> clazz =
                    conf.getClass(YarnConfiguration.NM_CONTAINER_MON_RESOURCE_CALCULATOR,
                        conf.getClass(
                            YarnConfiguration.NM_MON_RESOURCE_CALCULATOR, null,
                            ResourceCalculatorPlugin.class),
                        ResourceCalculatorPlugin.class);
            
          3. yarn-default.xml needs to be updated for the CONTAINER_MON configs we are changing the behavior of, and any default values there should be removed.
          Show
          kasha Karthik Kambatla added a comment - Hopefully, last set of comments: We should also not deprecate NM_CONTAINER_MON_RESOURCE_CALCULATOR. Should have been more explicit in my previous comment. ContainersMonitorImpl should prioritize container-specific configs and fallback if they are not present. Something along these lines: this .monitoringInterval = conf.getLong(YarnConfiguration.NM_CONTAINER_MON_INTERVAL_MS, conf.getLong(YarnConfiguration.NM_RESOURCE_MON_INTERVAL_MS, YarnConfiguration.DEFAULT_NM_RESOURCE_MON_INTERVAL_MS)); Class <? extends ResourceCalculatorPlugin> clazz = conf.getClass(YarnConfiguration.NM_CONTAINER_MON_RESOURCE_CALCULATOR, conf.getClass( YarnConfiguration.NM_MON_RESOURCE_CALCULATOR, null , ResourceCalculatorPlugin.class), ResourceCalculatorPlugin.class); yarn-default.xml needs to be updated for the CONTAINER_MON configs we are changing the behavior of, and any default values there should be removed.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 36s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 37s There were no new javac warning messages.
          +1 javadoc 9m 38s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 48s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          -1 checkstyle 2m 33s The applied patch generated 2 new checkstyle issues (total was 28, now 27).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 20s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.
          +1 yarn tests 6m 10s Tests passed in hadoop-yarn-server-nodemanager.
              53m 36s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12748290/YARN-3534-16.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d311a38
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8747/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8747/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 36s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 37s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 48s The applied patch generated 1 new checkstyle issues (total was 211, now 211). -1 checkstyle 2m 33s The applied patch generated 2 new checkstyle issues (total was 28, now 27). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common. +1 yarn tests 6m 10s Tests passed in hadoop-yarn-server-nodemanager.     53m 36s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748290/YARN-3534-16.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d311a38 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8747/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8747/console This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Fixing checkstyle

          Show
          elgoiri Íñigo Goiri added a comment - Fixing checkstyle
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 46s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 40s There were no new javac warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 46s The applied patch generated 4 new checkstyle issues (total was 211, now 214).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 23s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 20s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.
          +1 yarn tests 6m 9s Tests passed in hadoop-yarn-server-nodemanager.
              53m 43s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12748272/YARN-3534-16.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d311a38
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8742/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8742/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8742/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8742/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8742/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8742/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 46s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 40s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 46s The applied patch generated 4 new checkstyle issues (total was 211, now 214). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 23s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 20s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common. +1 yarn tests 6m 9s Tests passed in hadoop-yarn-server-nodemanager.     53m 43s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748272/YARN-3534-16.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d311a38 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8742/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8742/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8742/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8742/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8742/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8742/console This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          I guess in #2 you meant yarn-default.xml, right?
          I left #4 as it was but I'm not sure if you were referring to leave it as it was in trunk; let me know. The fallback is kind of complicated; we can give it a try if you want.

          Show
          elgoiri Íñigo Goiri added a comment - I guess in #2 you meant yarn-default.xml, right? I left #4 as it was but I'm not sure if you were referring to leave it as it was in trunk; let me know. The fallback is kind of complicated; we can give it a try if you want.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Solved most of Karthik Kambatla's comments.

          Show
          elgoiri Íñigo Goiri added a comment - Solved most of Karthik Kambatla 's comments.
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks for updating the patch, Íñigo Goiri. The latest patch looks pretty good. Few minor comments:

          1. Let us not deprecate NM_CONTAINER_MON_INTERVAL_MS. Deprecating the default value seems reasonable. One might want to specify a different interval for node monitoring and container monitoring.
          2. Make corresponding changes to yarn-site.xml
          3. NodeResourceMonitorImpl#serviceStop: the log message should say couldn't wait for the thread to join, and may be we should log it as WARN instead of INFO
          4. ContainersMonitorImpl: Users could continue to use the deprecated configs. If those are specified and the new ones are not, we should fallback to the deprecated ones. If this becomes too complicated, I am comfortable with leaving the configs for container-monitoring as is. We could file a follow-up JIRA to clean these configs for Hadoop-3.x.
          Show
          kasha Karthik Kambatla added a comment - Thanks for updating the patch, Íñigo Goiri . The latest patch looks pretty good. Few minor comments: Let us not deprecate NM_CONTAINER_MON_INTERVAL_MS . Deprecating the default value seems reasonable. One might want to specify a different interval for node monitoring and container monitoring. Make corresponding changes to yarn-site.xml NodeResourceMonitorImpl#serviceStop : the log message should say couldn't wait for the thread to join, and may be we should log it as WARN instead of INFO ContainersMonitorImpl: Users could continue to use the deprecated configs. If those are specified and the new ones are not, we should fallback to the deprecated ones. If this becomes too complicated, I am comfortable with leaving the configs for container-monitoring as is. We could file a follow-up JIRA to clean these configs for Hadoop-3.x.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 26s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 7m 46s The applied patch generated 2 additional warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 19s The applied patch generated 5 new checkstyle issues (total was 211, now 215).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 24s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api.
          -1 yarn tests 6m 6s Tests failed in hadoop-yarn-server-nodemanager.
              48m 0s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
            hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService
            hadoop.yarn.server.nodemanager.containermanager.container.TestContainer
            hadoop.yarn.server.nodemanager.TestDeletionService



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12744609/YARN-3534-15.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 1a0752d
          javac https://builds.apache.org/job/PreCommit-YARN-Build/8488/artifact/patchprocess/diffJavacWarnings.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8488/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8488/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8488/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8488/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8488/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 26s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 7m 46s The applied patch generated 2 additional warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 19s The applied patch generated 5 new checkstyle issues (total was 211, now 215). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 24s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api. -1 yarn tests 6m 6s Tests failed in hadoop-yarn-server-nodemanager.     48m 0s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService   hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService   hadoop.yarn.server.nodemanager.containermanager.container.TestContainer   hadoop.yarn.server.nodemanager.TestDeletionService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744609/YARN-3534-15.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1a0752d javac https://builds.apache.org/job/PreCommit-YARN-Build/8488/artifact/patchprocess/diffJavacWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8488/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8488/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8488/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8488/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8488/console This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Fixed issue with constants.
          We need to agree that these names are OK and that we are fine with deprecating the old ones.

          Show
          elgoiri Íñigo Goiri added a comment - Fixed issue with constants. We need to agree that these names are OK and that we are fine with deprecating the old ones.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 13s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 2m 59s The patch appears to cause the build to fail.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12744528/YARN-3534-14.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / ac60483
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8483/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 13s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 2m 59s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744528/YARN-3534-14.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ac60483 Console output https://builds.apache.org/job/PreCommit-YARN-Build/8483/console This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Updated to trunk (using ResourceUtilization already there).

          Show
          elgoiri Íñigo Goiri added a comment - Updated to trunk (using ResourceUtilization already there).
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 30s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 34s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          -1 release audit 0m 19s The applied patch generated 1 release audit warnings.
          -1 checkstyle 1m 41s The applied patch generated 1 new checkstyle issues (total was 213, now 212).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 20s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.
          +1 yarn tests 6m 3s Tests passed in hadoop-yarn-server-nodemanager.
              53m 3s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12739505/YARN-3534-12.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / b8341f1
          Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8247/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8247/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 30s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. -1 release audit 0m 19s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 41s The applied patch generated 1 new checkstyle issues (total was 213, now 212). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 20s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common. +1 yarn tests 6m 3s Tests passed in hadoop-yarn-server-nodemanager.     53m 3s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739505/YARN-3534-12.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b8341f1 Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8247/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8247/console This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Added license for ResourceUtilizations and extra comments.

          Show
          elgoiri Íñigo Goiri added a comment - Added license for ResourceUtilizations and extra comments.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 18s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 32s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          -1 release audit 0m 18s The applied patch generated 1 release audit warnings.
          -1 checkstyle 1m 44s The applied patch generated 1 new checkstyle issues (total was 212, now 211).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 21s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.
          +1 yarn tests 6m 13s Tests passed in hadoop-yarn-server-nodemanager.
              52m 52s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12737322/YARN-3534-11.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c59e745
          Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8180/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8180/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 18s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. -1 release audit 0m 18s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 44s The applied patch generated 1 new checkstyle issues (total was 212, now 211). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 21s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common. +1 yarn tests 6m 13s Tests passed in hadoop-yarn-server-nodemanager.     52m 52s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12737322/YARN-3534-11.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c59e745 Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8180/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8180/console This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Fixing build issues

          Show
          elgoiri Íñigo Goiri added a comment - Fixing build issues
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 16m 42s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 7m 34s The applied patch generated 1 additional warning messages.
          -1 javadoc 9m 39s The applied patch generated 3 additional warning messages.
          -1 release audit 0m 18s The applied patch generated 1 release audit warnings.
          -1 checkstyle 1m 24s The applied patch generated 9 new checkstyle issues (total was 212, now 220).
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 18s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 21s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.
          +1 yarn tests 6m 4s Tests passed in hadoop-yarn-server-nodemanager.
              50m 53s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12737047/YARN-3534-10.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c59e745
          javac https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/diffJavacWarnings.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/diffJavadocWarnings.txt
          Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/whitespace.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8176/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8176/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 16m 42s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 7m 34s The applied patch generated 1 additional warning messages. -1 javadoc 9m 39s The applied patch generated 3 additional warning messages. -1 release audit 0m 18s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 24s The applied patch generated 9 new checkstyle issues (total was 212, now 220). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 18s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 21s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common. +1 yarn tests 6m 4s Tests passed in hadoop-yarn-server-nodemanager.     50m 53s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12737047/YARN-3534-10.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c59e745 javac https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/diffJavacWarnings.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/diffJavadocWarnings.txt Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/whitespace.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8176/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8176/console This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Solved some comments

          Show
          elgoiri Íñigo Goiri added a comment - Solved some comments
          Hide
          elgoiri Íñigo Goiri added a comment -

          I'll do the changes you propose but I have a quesiton about 1: right now, I don't have ResourceUtilization in thos patch, should I added to this one? I wouldn't add any of the PBImpl stuff here.

          Show
          elgoiri Íñigo Goiri added a comment - I'll do the changes you propose but I have a quesiton about 1: right now, I don't have ResourceUtilization in thos patch, should I added to this one? I wouldn't add any of the PBImpl stuff here.
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks for working on this, Inigo.

          Few comments:

          1. Given these stats are to be sent to the RM in heartbeat, should we capture the aggregate node resource usage in ResourceUtilization?
          2. Instead of adding a separate DEFAULT_NM_NODE_MON_INTERVAL_MS, we should probably just re-use the default for container-monitor?
          3. Also, should we add another config yarn.nodemanager.usage-monitor.interval-ms that both container-monitor and node-monitor inherit unless specified otherwise? If that seems reasonable, we should deprecate the default value for container-monitor-interval.
          4. For the monitoring thread,
            1. set the thread name?
            2. make it a daemon thread?
            3. on monitoringThread.join(), specify a timeout as well.
            4. in the corresponding catch-block, at least log that we couldn't wait until the monitoring-thread is interrupted.
          Show
          kasha Karthik Kambatla added a comment - Thanks for working on this, Inigo. Few comments: Given these stats are to be sent to the RM in heartbeat, should we capture the aggregate node resource usage in ResourceUtilization ? Instead of adding a separate DEFAULT_NM_NODE_MON_INTERVAL_MS , we should probably just re-use the default for container-monitor? Also, should we add another config yarn.nodemanager.usage-monitor.interval-ms that both container-monitor and node-monitor inherit unless specified otherwise? If that seems reasonable, we should deprecate the default value for container-monitor-interval. For the monitoring thread, set the thread name? make it a daemon thread? on monitoringThread.join() , specify a timeout as well. in the corresponding catch-block, at least log that we couldn't wait until the monitoring-thread is interrupted.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Where do you want to put the vmem and where to get the cpu-wall time?
          Regarding YarnConfiguration, I didn't understand what you wanted to do there. Do you want to put the constants in the class itself?

          Show
          elgoiri Íñigo Goiri added a comment - Where do you want to put the vmem and where to get the cpu-wall time? Regarding YarnConfiguration, I didn't understand what you wanted to do there. Do you want to put the constants in the class itself?
          Hide
          elgoiri Íñigo Goiri added a comment -

          Removed nodemanager context.
          Added vmem (I cannot put it inside the Resource though).

          Show
          elgoiri Íñigo Goiri added a comment - Removed nodemanager context. Added vmem (I cannot put it inside the Resource though).
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          The argument of how to pass this information to the RM seems still open. Should I open a new JIRA for that and move the discussion about using REST, heartbeat etc to that one? Or should we have that discussion in one of the related JIRAs (e.g., YARN-3332)?

          I believe there are existing tickets for this. I started consolidating all the related efforts, let me do that so that you can start working on this once I identify the right JIRA.

          Any proposal for unit tests? The ones in ContainerMonitorImpl don't really apply.

          You can reuse some of the code at TestLinuxResourceCalculatorPlugin and write a new TestNodeResourceMonitor that measures utilization against mock information.

          Couple other comments:

          • context variable is unused
          • Can we also capture vmem usage as well as cpu-wall time?
          • As per my proposal at YARN-3332, I'd like to refactor this later to be the statistics-agent. So, for now, can you just move the configuration-properties from YarnConfiguration into NodeResourceMonitorImpl?
          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - The argument of how to pass this information to the RM seems still open. Should I open a new JIRA for that and move the discussion about using REST, heartbeat etc to that one? Or should we have that discussion in one of the related JIRAs (e.g., YARN-3332 )? I believe there are existing tickets for this. I started consolidating all the related efforts, let me do that so that you can start working on this once I identify the right JIRA. Any proposal for unit tests? The ones in ContainerMonitorImpl don't really apply. You can reuse some of the code at TestLinuxResourceCalculatorPlugin and write a new TestNodeResourceMonitor that measures utilization against mock information. Couple other comments: context variable is unused Can we also capture vmem usage as well as cpu-wall time? As per my proposal at YARN-3332 , I'd like to refactor this later to be the statistics-agent. So, for now, can you just move the configuration-properties from YarnConfiguration into NodeResourceMonitorImpl?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 39s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 javadoc 9m 30s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 28s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 23s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 yarn tests 0m 30s Tests passed in hadoop-yarn-api.
          +1 yarn tests 5m 51s Tests passed in hadoop-yarn-server-nodemanager.
              44m 26s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12729911/YARN-3534-8.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f1a152c
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7594/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7594/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7594/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7594/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 39s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 30s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 28s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 23s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 0m 30s Tests passed in hadoop-yarn-api. +1 yarn tests 5m 51s Tests passed in hadoop-yarn-server-nodemanager.     44m 26s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729911/YARN-3534-8.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f1a152c hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7594/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7594/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7594/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7594/console This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Fixing code style issues (nwo the output of the checker is meaningful so life is much easier).

          The broken tests weren't related to my changes.

          Any proposal for unit tests? The ones in ContainerMonitorImpl don't really apply.

          Show
          elgoiri Íñigo Goiri added a comment - Fixing code style issues (nwo the output of the checker is meaningful so life is much easier). The broken tests weren't related to my changes. Any proposal for unit tests? The ones in ContainerMonitorImpl don't really apply.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 42s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 javadoc 9m 36s There were no new javadoc warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 22s The applied patch generated 2 new checkstyle issues (total was 211, now 213).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 2m 24s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          -1 yarn tests 5m 48s Tests failed in hadoop-yarn-server-nodemanager.
              44m 33s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12729860/YARN-3534-7.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6f541ed
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7589/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7589/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7589/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7589/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7589/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 42s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 22s The applied patch generated 2 new checkstyle issues (total was 211, now 213). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 24s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. -1 yarn tests 5m 48s Tests failed in hadoop-yarn-server-nodemanager.     44m 33s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729860/YARN-3534-7.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6f541ed checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7589/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7589/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7589/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7589/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7589/console This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Trying to trigger built from latest trunk

          Show
          elgoiri Íñigo Goiri added a comment - Trying to trigger built from latest trunk
          Hide
          hadoopqa Hadoop QA added a comment -

          The patch artifact directory has been removed!
          This is a fatal error for test-patch.sh. Aborting.
          Jenkins (node H4) information at https://builds.apache.org/job/PreCommit-YARN-Build/7569/ may provide some hints.

          Show
          hadoopqa Hadoop QA added a comment - The patch artifact directory has been removed! This is a fatal error for test-patch.sh. Aborting. Jenkins (node H4) information at https://builds.apache.org/job/PreCommit-YARN-Build/7569/ may provide some hints.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Sorry about that, I did the diff and then I did the patch 10 minutes after... meanwhile there were change sin trunk and it got all messed up. I think this one should be OK. No unit tests though.

          Show
          elgoiri Íñigo Goiri added a comment - Sorry about that, I did the diff and then I did the patch 10 minutes after... meanwhile there were change sin trunk and it got all messed up. I think this one should be OK. No unit tests though.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Diff to the latest trunk.

          Show
          elgoiri Íñigo Goiri added a comment - Diff to the latest trunk.
          Hide
          vvasudev Varun Vasudev added a comment -

          Íñigo Goiri - the latest patch(YARN-3534-5.patch) has modifications to the checkstyle script as well as OsSecureRandom.java. I assume those changes are not meant to be part of your patch. Can you please upload a patch with just your changes? Thanks!

          Show
          vvasudev Varun Vasudev added a comment - Íñigo Goiri - the latest patch( YARN-3534 -5.patch) has modifications to the checkstyle script as well as OsSecureRandom.java. I assume those changes are not meant to be part of your patch. Can you please upload a patch with just your changes? Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 14m 45s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 javac 6m 29s There were no new javac warning messages.
          +1 javadoc 7m 25s There were no new javadoc warning messages.
          -1 release audit 22m 41s The applied patch generated 364 release audit warnings.
          -1 checkstyle 4m 26s The applied patch generated 2 additional checkstyle issues.
          0 shellcheck 4m 26s Shellcheck was not available.
          +1 install 1m 12s mvn install still works.
          +1 eclipse:eclipse 0m 30s The patch built with eclipse:eclipse.
          +1 findbugs 4m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 common tests 23m 7s Tests passed in hadoop-common.
          +1 yarn tests 0m 29s Tests passed in hadoop-yarn-api.
          -1 yarn tests 5m 47s Tests failed in hadoop-yarn-server-nodemanager.
              90m 58s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12729676/YARN-3534-5.patch
          Optional Tests shellcheck javadoc javac unit findbugs checkstyle
          git revision trunk / 98a6176
          Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/7564/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7564/artifact/patchprocess/checkstyle-result-diff.txt
          hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7564/artifact/patchprocess/testrun_hadoop-common.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7564/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7564/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7564/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7564/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 14m 45s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 6m 29s There were no new javac warning messages. +1 javadoc 7m 25s There were no new javadoc warning messages. -1 release audit 22m 41s The applied patch generated 364 release audit warnings. -1 checkstyle 4m 26s The applied patch generated 2 additional checkstyle issues. 0 shellcheck 4m 26s Shellcheck was not available. +1 install 1m 12s mvn install still works. +1 eclipse:eclipse 0m 30s The patch built with eclipse:eclipse. +1 findbugs 4m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 23m 7s Tests passed in hadoop-common. +1 yarn tests 0m 29s Tests passed in hadoop-yarn-api. -1 yarn tests 5m 47s Tests failed in hadoop-yarn-server-nodemanager.     90m 58s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729676/YARN-3534-5.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / 98a6176 Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/7564/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7564/artifact/patchprocess/checkstyle-result-diff.txt hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7564/artifact/patchprocess/testrun_hadoop-common.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7564/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7564/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7564/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7564/console This message was automatically generated.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Converting this to be a sub-task of YARN-1011. See my last comment at YARN-3481.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Converting this to be a sub-task of YARN-1011 . See my last comment at YARN-3481 .
          Hide
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-YARN-Build/7564/console in case of problems.

          Show
          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-YARN-Build/7564/console in case of problems.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Fixed compilation issue with the interface.

          Show
          elgoiri Íñigo Goiri added a comment - Fixed compilation issue with the interface.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 33s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          -1 javac 1m 48s The patch appears to cause the build to fail.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12729617/YARN-3534-4.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f0db797
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7560/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 33s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 whitespace 0m 0s The patch has no lines that end in whitespace. -1 javac 1m 48s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729617/YARN-3534-4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f0db797 Console output https://builds.apache.org/job/PreCommit-YARN-Build/7560/console This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Isolating the changes for the NodeResourceMonitorImpl. The aggregationg of data and sending to the RM will be done in another JIRA.

          Show
          elgoiri Íñigo Goiri added a comment - Isolating the changes for the NodeResourceMonitorImpl. The aggregationg of data and sending to the RM will be done in another JIRA.
          Hide
          elgoiri Íñigo Goiri added a comment -

          It looks like there's a consensus on splitting on monitoring and reporting. If everybody is cool with it, I will rename this JIRA to "Collect node resource utilization" and just do the implementaiton of NodeResourceMonitor.

          The argument of how to pass this information to the RM seems still open. Should I open a new JIRA for that and move the discussion about using REST, heartbeat etc to that one? Or should we have that discussion in one of the related JIRAs (e.g., YARN-3332)?

          Show
          elgoiri Íñigo Goiri added a comment - It looks like there's a consensus on splitting on monitoring and reporting. If everybody is cool with it, I will rename this JIRA to "Collect node resource utilization" and just do the implementaiton of NodeResourceMonitor. The argument of how to pass this information to the RM seems still open. Should I open a new JIRA for that and move the discussion about using REST, heartbeat etc to that one? Or should we have that discussion in one of the related JIRAs (e.g., YARN-3332 )?
          Hide
          kasha Karthik Kambatla added a comment -

          I'm not sure if we should use RPC to expose the stats.

          As YARN-3332 suggests, we should expose these stats through REST. However, for the RM (scheduler) to get this information, I think RPC (through node heartbeat) is better. That way, the RM doesn't have to poll the REST API for every node there is an NM heartbeat for.

          If the suggestion is for the NM to get this information from the REST API, I think that is fine. However, since this communication is private to NM, we can always change this later once YARN-3332 is ready.

          Show
          kasha Karthik Kambatla added a comment - I'm not sure if we should use RPC to expose the stats. As YARN-3332 suggests, we should expose these stats through REST. However, for the RM (scheduler) to get this information, I think RPC (through node heartbeat) is better. That way, the RM doesn't have to poll the REST API for every node there is an NM heartbeat for. If the suggestion is for the NM to get this information from the REST API, I think that is fine. However, since this communication is private to NM, we can always change this later once YARN-3332 is ready.
          Hide
          vvasudev Varun Vasudev added a comment -

          Thanks for the path Íñigo Goiri! Is it possible for you to split up the patch into two - one to record the memory and cpu utilization in NodeResourceMonitorImpl and one for the RPC changes. If you could file a separate JIRA for the NodeResourceMonitorImpl changes that you have made(they are definitely required and should go in), that would be ideal.

          I'm not sure if we should use RPC to expose the stats. Using RPC restricts the stats to a limited number of services whereas using REST lets a larger number of services access the stats - which is why YARN-3332 wants to use REST to expose the NM stats. Using REST should also make it easier to add more stats with respect to utilization.

          Show
          vvasudev Varun Vasudev added a comment - Thanks for the path Íñigo Goiri ! Is it possible for you to split up the patch into two - one to record the memory and cpu utilization in NodeResourceMonitorImpl and one for the RPC changes. If you could file a separate JIRA for the NodeResourceMonitorImpl changes that you have made(they are definitely required and should go in), that would be ideal. I'm not sure if we should use RPC to expose the stats. Using RPC restricts the stats to a limited number of services whereas using REST lets a larger number of services access the stats - which is why YARN-3332 wants to use REST to expose the NM stats. Using REST should also make it easier to add more stats with respect to utilization.
          Hide
          elgoiri Íñigo Goiri added a comment -

          I could first do the NodeResourceMonitorImpl and then add that info it to the heartbeat in a second JIRA. However, I thought the addition to the heartbeat was simple enough. Anyways, up to you; I can split it in two.

          Regarding the extra ResourceUtilizations layers, I could get rid of them for now as it's not currently needed. However, in parallel, I'm privately working on a scheduler which is aware of utilization and it's much better to use those layers as I need the floats for the CPUs instead of plain int. I thought about making ResourceUtilizaiton an extension of Resource but it does not work...

          Anyway, I'll send a patch without the extra ResourceUtilizations layer as it's not really required for this patch. Does that sound reasonable?

          Show
          elgoiri Íñigo Goiri added a comment - I could first do the NodeResourceMonitorImpl and then add that info it to the heartbeat in a second JIRA. However, I thought the addition to the heartbeat was simple enough. Anyways, up to you; I can split it in two. Regarding the extra ResourceUtilizations layers, I could get rid of them for now as it's not currently needed. However, in parallel, I'm privately working on a scheduler which is aware of utilization and it's much better to use those layers as I need the floats for the CPUs instead of plain int. I thought about making ResourceUtilizaiton an extension of Resource but it does not work... Anyway, I'll send a patch without the extra ResourceUtilizations layer as it's not really required for this patch. Does that sound reasonable?
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Please see my comment on YARN-3481.

          If nothing, we should separate the reporting bits from the measurement bit?

          Also, we don't need the ResourceUtilizationCalculator layers? Once the information reaches the ResourceManager, RM can use the regular Resources utils.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Please see my comment on YARN-3481 . If nothing, we should separate the reporting bits from the measurement bit? Also, we don't need the ResourceUtilizationCalculator layers? Once the information reaches the ResourceManager, RM can use the regular Resources utils.
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks for pointing me to YARN-3481, Inigo. Didn't see the other JIRA.

          Reading through those comments, I agree it might be better to add a separate class/proto for ResourceUtilization. Efforts like Tetris could add disk/network utilization even if the API itself doesn't support them explicitly.

          Show
          kasha Karthik Kambatla added a comment - Thanks for pointing me to YARN-3481 , Inigo. Didn't see the other JIRA. Reading through those comments, I agree it might be better to add a separate class/proto for ResourceUtilization. Efforts like Tetris could add disk/network utilization even if the API itself doesn't support them explicitly.
          Hide
          elgoiri Íñigo Goiri added a comment -

          The original reason for having ResourceUtilization was getting better granularity for the CPUs. We had some discussion about it in YARN-3481; take a look there and chim in.

          My original implementation had Resource but when trying to do scheduling of containers based on this, there were a lot of holes in the scheduling. Given this, I thought this patch was a good place to create this new utilization entity with CPU as a float.

          Regarding the metrics in the NodeResourceMonitor, I completely agree. I thought about doing it right away but as you mentioned, it seemed a better idea to save it for another JIRA. Let's do that.

          Show
          elgoiri Íñigo Goiri added a comment - The original reason for having ResourceUtilization was getting better granularity for the CPUs. We had some discussion about it in YARN-3481 ; take a look there and chim in. My original implementation had Resource but when trying to do scheduling of containers based on this, there were a lot of holes in the scheduling. Given this, I thought this patch was a good place to create this new utilization entity with CPU as a float. Regarding the metrics in the NodeResourceMonitor, I completely agree. I thought about doing it right away but as you mentioned, it seemed a better idea to save it for another JIRA. Let's do that.
          Hide
          kasha Karthik Kambatla added a comment -

          I notice the patch tries to provide utilization information in bytes for memory and float for CPU. Since the RM schedules in MB and vcores, seeing the utilization as rounded-up values in a Resource object is probably enough.

          Show
          kasha Karthik Kambatla added a comment - I notice the patch tries to provide utilization information in bytes for memory and float for CPU. Since the RM schedules in MB and vcores, seeing the utilization as rounded-up values in a Resource object is probably enough.
          Hide
          kasha Karthik Kambatla added a comment -

          Skimmed through the latest patch. High-level comments/questions:

          1. Do we need a separate class/proto for ResourceUtilization? Could we just reuse Resource? That should make the patch significantly small.
          2. Would be nice to have NodeResourceMonitor emit metrics for usage. We could do this on a follow-up JIRA.
          Show
          kasha Karthik Kambatla added a comment - Skimmed through the latest patch. High-level comments/questions: Do we need a separate class/proto for ResourceUtilization? Could we just reuse Resource? That should make the patch significantly small. Would be nice to have NodeResourceMonitor emit metrics for usage. We could do this on a follow-up JIRA.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 38s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 javac 7m 29s There were no new javac warning messages.
          +1 javadoc 9m 34s There were no new javadoc warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 7m 47s The applied patch generated 11 additional checkstyle issues.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 37s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 yarn tests 0m 29s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 1s Tests passed in hadoop-yarn-common.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 6m 5s Tests passed in hadoop-yarn-server-nodemanager.
              55m 33s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12727786/YARN-3534-3.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / bcf89dd
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7489/artifact/patchprocess/checkstyle-result-diff.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7489/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7489/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7489/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7489/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7489/testReport/
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7489/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 38s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 29s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 7m 47s The applied patch generated 11 additional checkstyle issues. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 37s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 0m 29s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 1s Tests passed in hadoop-yarn-common. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common. +1 yarn tests 6m 5s Tests passed in hadoop-yarn-server-nodemanager.     55m 33s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12727786/YARN-3534-3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / bcf89dd checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7489/artifact/patchprocess/checkstyle-result-diff.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7489/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7489/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7489/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7489/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7489/testReport/ Console output https://builds.apache.org/job/PreCommit-YARN-Build/7489/console This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Updating trunk before it goes ahead.

          Show
          elgoiri Íñigo Goiri added a comment - Updating trunk before it goes ahead.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Removing spaces at the end of the line; Jenkins doesn't seem to like them anymore.

          Show
          elgoiri Íñigo Goiri added a comment - Removing spaces at the end of the line; Jenkins doesn't seem to like them anymore.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 35s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          -1 whitespace 0m 0s The patch has 78 line(s) that end in whitespace.
          +1 javac 7m 29s There were no new javac warning messages.
          +1 javadoc 9m 34s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 5m 32s The applied patch generated 11 additional checkstyle issues.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 yarn tests 0m 21s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 1s Tests passed in hadoop-yarn-common.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 5m 55s Tests passed in hadoop-yarn-server-nodemanager.
              53m 2s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12727766/YARN-3534-2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0b3f895
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/whitespace.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/checkstyle-result-diff.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7485/testReport/
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7485/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 35s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 whitespace 0m 0s The patch has 78 line(s) that end in whitespace. +1 javac 7m 29s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 5m 32s The applied patch generated 11 additional checkstyle issues. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 0m 21s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 1s Tests passed in hadoop-yarn-common. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common. +1 yarn tests 5m 55s Tests passed in hadoop-yarn-server-nodemanager.     53m 2s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12727766/YARN-3534-2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0b3f895 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/whitespace.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/checkstyle-result-diff.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7485/testReport/ Console output https://builds.apache.org/job/PreCommit-YARN-Build/7485/console This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Merged to trunk.

          Show
          elgoiri Íñigo Goiri added a comment - Merged to trunk.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12727762/YARN-3534-1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0b3f895
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7484/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12727762/YARN-3534-1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0b3f895 Console output https://builds.apache.org/job/PreCommit-YARN-Build/7484/console This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          No unit tests yet.

          Show
          elgoiri Íñigo Goiri added a comment - No unit tests yet.
          Hide
          elgoiri Íñigo Goiri added a comment -

          This does not include unit tests yet. Working on them.

          Show
          elgoiri Íñigo Goiri added a comment - This does not include unit tests yet. Working on them.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Sunil G, my plan is to send the same entity as in YARN-3482: ResourceUtilization. For now, it will have memory in MB and Virtual Cores as a float. We can extend this for other resources.
          In terms of the collection of the utilization, this will be monitored by the NodeResourceMonitor (currently empty) which will mimic the behavior of the ContainerMonitor which uses two configuration parameters: (1) if we monitor the resources of the node and (2) how often the resources are monitored.

          I'm in the last stages of testing it and tomorrow I'll post a patch with this change. The structure was already there so it's a matter of adding the communication part..

          Show
          elgoiri Íñigo Goiri added a comment - Sunil G , my plan is to send the same entity as in YARN-3482 : ResourceUtilization. For now, it will have memory in MB and Virtual Cores as a float. We can extend this for other resources. In terms of the collection of the utilization, this will be monitored by the NodeResourceMonitor (currently empty) which will mimic the behavior of the ContainerMonitor which uses two configuration parameters: (1) if we monitor the resources of the node and (2) how often the resources are monitored. I'm in the last stages of testing it and tomorrow I'll post a patch with this change. The structure was already there so it's a matter of adding the communication part..
          Hide
          sunilg Sunil G added a comment -

          resource utilization of the nodes

          Is this going to be configurable item to look for, or all resources will be monitored?

          Show
          sunilg Sunil G added a comment - resource utilization of the nodes Is this going to be configurable item to look for, or all resources will be monitored?

            People

            • Assignee:
              elgoiri Íñigo Goiri
              Reporter:
              elgoiri Íñigo Goiri
            • Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 336h
                336h
                Remaining:
                Remaining Estimate - 336h
                336h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development