Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: nodemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    1. YARN-1012-1.patch
      14 kB
      Íñigo Goiri
    2. YARN-1012-10.patch
      24 kB
      Íñigo Goiri
    3. YARN-1012-11.patch
      26 kB
      Íñigo Goiri
    4. YARN-1012-2.patch
      14 kB
      Íñigo Goiri
    5. YARN-1012-3.patch
      14 kB
      Íñigo Goiri
    6. YARN-1012-4.patch
      20 kB
      Íñigo Goiri
    7. YARN-1012-5.patch
      19 kB
      Íñigo Goiri
    8. YARN-1012-6.patch
      19 kB
      Íñigo Goiri
    9. YARN-1012-7.patch
      22 kB
      Íñigo Goiri
    10. YARN-1012-8.patch
      30 kB
      Íñigo Goiri
    11. YARN-1012-9.patch
      24 kB
      Íñigo Goiri

      Issue Links

        Activity

        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        YARN-3481 is closed as dup of this JIRA.

        Assigning this one to Íñigo Goiri who was moving forward on that JIRA.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - YARN-3481 is closed as dup of this JIRA. Assigning this one to Íñigo Goiri who was moving forward on that JIRA.
        Hide
        elgoiri Íñigo Goiri added a comment -

        Thanks for adjusting the JIRAs Vinod Kumar Vavilapalli.

        My proposal is to get the data from the ContainerMonitorImpl, aggregate it and put it in a new entity called ResourceUtilization which will contain for now CPU as a float and memory. Then, this will go into in the NodeHealthStauts.

        The other option is to add the utilization per container using the ResourceUtilization again but adding it per each container into the ContainerStatus.

        Both of them would require the new ResourceUtilization which in this case for simplicity will just be it without the whole ResourceUtilizations layer.

        Which one you guys prefer?

        Show
        elgoiri Íñigo Goiri added a comment - Thanks for adjusting the JIRAs Vinod Kumar Vavilapalli . My proposal is to get the data from the ContainerMonitorImpl, aggregate it and put it in a new entity called ResourceUtilization which will contain for now CPU as a float and memory. Then, this will go into in the NodeHealthStauts. The other option is to add the utilization per container using the ResourceUtilization again but adding it per each container into the ContainerStatus. Both of them would require the new ResourceUtilization which in this case for simplicity will just be it without the whole ResourceUtilizations layer. Which one you guys prefer?
        Hide
        elgoiri Íñigo Goiri added a comment -

        Adds container utilization to the container status sent to the RM in the heartbeat.

        Show
        elgoiri Íñigo Goiri added a comment - Adds container utilization to the container status sent to the RM in the heartbeat.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 42s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        -1 javac 2m 22s The patch appears to cause the build to fail.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12731013/YARN-1012-1.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 31b627b
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/7745/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 42s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 2m 22s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731013/YARN-1012-1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 31b627b Console output https://builds.apache.org/job/PreCommit-YARN-Build/7745/console This message was automatically generated.
        Hide
        elgoiri Íñigo Goiri added a comment -

        Patch against latest trunk to see if it fixes the compilation errors.

        Show
        elgoiri Íñigo Goiri added a comment - Patch against latest trunk to see if it fixes the compilation errors.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 39s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        -1 javac 2m 22s The patch appears to cause the build to fail.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12731026/YARN-1012-2.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 4c7b9b6
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/7746/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 39s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 2m 22s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731026/YARN-1012-2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4c7b9b6 Console output https://builds.apache.org/job/PreCommit-YARN-Build/7746/console This message was automatically generated.
        Hide
        elgoiri Íñigo Goiri added a comment -

        Trying with the latest trunk.

        Show
        elgoiri Íñigo Goiri added a comment - Trying with the latest trunk.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 15m 17s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        -1 javac 2m 23s The patch appears to cause the build to fail.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12733899/YARN-1012-3.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 8860e35
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8004/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 17s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 2m 23s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733899/YARN-1012-3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8860e35 Console output https://builds.apache.org/job/PreCommit-YARN-Build/8004/console This message was automatically generated.
        Hide
        kasha Karthik Kambatla added a comment -

        Looks like the patch is missing ResourceUtilizationPBImpl, and hence doesn't build. Could you please include those new files as well?

        Comments on the patch itself:

        1. Given this is all new code, let us hold off on exposing it to end users just yet. Can we mark ContainerStatus#getUtilization Public-Unstable?
        2. Is there a reason folks would want to turn off tracking utilization? If not, let us get rid of the config and always track it?
        3. When logging at debug level, we want to check if debug logging is enabled to avoid string creation and concat.
        4. I notice that we are using a float for virtual_cores. Do we anticipate using this value in any calculations? If yes, should we change this to be millivcores and int instead to avoid those floating point operations. Given this is just tracking utilization, I suspect we ll do any calculations.
        5. In ContainerMonitorsImpl, we save utilization and then set container metrics. Should we leave this as is? Or, link them up so that the ContainerMonitorsImpl is aware of only one of them?
        Show
        kasha Karthik Kambatla added a comment - Looks like the patch is missing ResourceUtilizationPBImpl, and hence doesn't build. Could you please include those new files as well? Comments on the patch itself: Given this is all new code, let us hold off on exposing it to end users just yet. Can we mark ContainerStatus#getUtilization Public-Unstable? Is there a reason folks would want to turn off tracking utilization? If not, let us get rid of the config and always track it? When logging at debug level, we want to check if debug logging is enabled to avoid string creation and concat. I notice that we are using a float for virtual_cores. Do we anticipate using this value in any calculations? If yes, should we change this to be millivcores and int instead to avoid those floating point operations. Given this is just tracking utilization, I suspect we ll do any calculations. In ContainerMonitorsImpl, we save utilization and then set container metrics. Should we leave this as is? Or, link them up so that the ContainerMonitorsImpl is aware of only one of them?
        Hide
        elgoiri Íñigo Goiri added a comment -

        I don't know how I missed the missing files... I've been checking this for days. Fixed now.

        Agreed and fixed 1, 2, 3, and 4.

        I don't know what to do with 5... your call.

        Show
        elgoiri Íñigo Goiri added a comment - I don't know how I missed the missing files... I've been checking this for days. Fixed now. Agreed and fixed 1, 2, 3, and 4. I don't know what to do with 5... your call.
        Hide
        elgoiri Íñigo Goiri added a comment -

        Added missing files.
        Fixed some of the comments.

        Show
        elgoiri Íñigo Goiri added a comment - Added missing files. Fixed some of the comments.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 37s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 38s There were no new javac warning messages.
        +1 javadoc 9m 37s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 46s The applied patch generated 5 new checkstyle issues (total was 1, now 6).
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 3m 48s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 29s Tests passed in hadoop-yarn-api.
        -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common.
        +1 yarn tests 6m 7s Tests passed in hadoop-yarn-server-nodemanager.
            48m 55s  



        Reason Tests
        Failed unit tests hadoop.yarn.api.TestPBImplRecords



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12734340/YARN-1012-4.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / fb6b38d
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8038/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8038/artifact/patchprocess/whitespace.txt
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8038/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8038/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8038/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8038/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8038/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 37s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 38s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 46s The applied patch generated 5 new checkstyle issues (total was 1, now 6). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 48s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 29s Tests passed in hadoop-yarn-api. -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common. +1 yarn tests 6m 7s Tests passed in hadoop-yarn-server-nodemanager.     48m 55s   Reason Tests Failed unit tests hadoop.yarn.api.TestPBImplRecords Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12734340/YARN-1012-4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / fb6b38d checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8038/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8038/artifact/patchprocess/whitespace.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8038/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8038/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8038/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8038/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8038/console This message was automatically generated.
        Hide
        elgoiri Íñigo Goiri added a comment -

        Fixed style issues.

        Show
        elgoiri Íñigo Goiri added a comment - Fixed style issues.
        Hide
        elgoiri Íñigo Goiri added a comment -

        I understand now what you're saying in 5. I actually tried to store the values in the metrics and get it from there but the interface for the metrics is not easy. Ideas?

        Show
        elgoiri Íñigo Goiri added a comment - I understand now what you're saying in 5. I actually tried to store the values in the metrics and get it from there but the interface for the metrics is not easy. Ideas?
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 15m 10s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 46s There were no new javac warning messages.
        +1 javadoc 10m 0s There were no new javadoc warning messages.
        +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 2m 5s There were no new checkstyle issues.
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 3m 56s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
        -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common.
        +1 yarn tests 6m 17s Tests passed in hadoop-yarn-server-nodemanager.
            50m 4s  



        Reason Tests
        Failed unit tests hadoop.yarn.api.TestPBImplRecords



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12734640/YARN-1012-5.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 05e04f3
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8048/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8048/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8048/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8048/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8048/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 10s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 46s There were no new javac warning messages. +1 javadoc 10m 0s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 5s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 56s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common. +1 yarn tests 6m 17s Tests passed in hadoop-yarn-server-nodemanager.     50m 4s   Reason Tests Failed unit tests hadoop.yarn.api.TestPBImplRecords Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12734640/YARN-1012-5.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 05e04f3 hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8048/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8048/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8048/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8048/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8048/console This message was automatically generated.
        Hide
        kasha Karthik Kambatla added a comment -

        The test failure is related to the patch. Íñigo Goiri - can you take a look?

        Other comments:

        1. ResourceUtilization should be marked Unstable. Also, does it need to be Public? Accordingly, we need to update or get rid of annotations for the member methods.
        2. It would be nice to add a unit test to ensure resource-utilization is actually tracked when container-monitoring is enabled.
        Show
        kasha Karthik Kambatla added a comment - The test failure is related to the patch. Íñigo Goiri - can you take a look? Other comments: ResourceUtilization should be marked Unstable. Also, does it need to be Public? Accordingly, we need to update or get rid of annotations for the member methods. It would be nice to add a unit test to ensure resource-utilization is actually tracked when container-monitoring is enabled.
        Hide
        elgoiri Íñigo Goiri added a comment -

        I checked the issue with testContainerStatusPBImpl and I cannot figure out what's wrong there. Am I missing any method in ResourceUtilization?

        I also updated the interfaces and made it Unstable and Private (which I think matches our scope).

        Regarding the unit test, how would you check? Would you check to context.getContainers()? This related to your original quesiton of where should we store this information (ContainerMetrics or ContainerStatus).

        Show
        elgoiri Íñigo Goiri added a comment - I checked the issue with testContainerStatusPBImpl and I cannot figure out what's wrong there. Am I missing any method in ResourceUtilization? I also updated the interfaces and made it Unstable and Private (which I think matches our scope). Regarding the unit test, how would you check? Would you check to context.getContainers()? This related to your original quesiton of where should we store this information (ContainerMetrics or ContainerStatus).
        Hide
        elgoiri Íñigo Goiri added a comment -

        I checked the issue with testContainerStatusPBImpl and I cannot figure out what's wrong there. Am I missing any method in ResourceUtilization?

        I also updated the interfaces and made it Unstable and Private (which I think matches our scope).

        Regarding the unit test, how would you check? Would you check to context.getContainers()? This related to your original quesiton of where should we store this information (ContainerMetrics or ContainerStatus).

        Show
        elgoiri Íñigo Goiri added a comment - I checked the issue with testContainerStatusPBImpl and I cannot figure out what's wrong there. Am I missing any method in ResourceUtilization? I also updated the interfaces and made it Unstable and Private (which I think matches our scope). Regarding the unit test, how would you check? Would you check to context.getContainers()? This related to your original quesiton of where should we store this information (ContainerMetrics or ContainerStatus).
        Hide
        elgoiri Íñigo Goiri added a comment -

        Changed annotations for ResourceUtilization.

        Show
        elgoiri Íñigo Goiri added a comment - Changed annotations for ResourceUtilization.
        Hide
        elgoiri Íñigo Goiri added a comment -

        Adding missing ResourceUtilization to unit test.

        Show
        elgoiri Íñigo Goiri added a comment - Adding missing ResourceUtilization to unit test.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 43s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 36s There were no new javac warning messages.
        +1 javadoc 9m 38s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 2m 6s There were no new checkstyle issues.
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 3m 47s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api.
        -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common.
        +1 yarn tests 6m 18s Tests passed in hadoop-yarn-server-nodemanager.
            49m 6s  



        Reason Tests
        Failed unit tests hadoop.yarn.api.TestPBImplRecords



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12735350/YARN-1012-6.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 022f49d
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8087/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8087/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8087/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8087/testReport/
        Java 1.7.0_55
        uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8087/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 43s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 6s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 3m 47s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api. -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common. +1 yarn tests 6m 18s Tests passed in hadoop-yarn-server-nodemanager.     49m 6s   Reason Tests Failed unit tests hadoop.yarn.api.TestPBImplRecords Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735350/YARN-1012-6.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 022f49d hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8087/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8087/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8087/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8087/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8087/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 53s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 7m 35s There were no new javac warning messages.
        +1 javadoc 9m 35s There were no new javadoc warning messages.
        +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 2m 5s There were no new checkstyle issues.
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 3m 48s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 22s Tests passed in hadoop-yarn-api.
        +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common.
        +1 yarn tests 6m 17s Tests passed in hadoop-yarn-server-nodemanager.
            49m 10s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12735357/YARN-1012-7.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 022f49d
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8088/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8088/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8088/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8088/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8088/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 53s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 5s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 48s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 22s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common. +1 yarn tests 6m 17s Tests passed in hadoop-yarn-server-nodemanager.     49m 10s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735357/YARN-1012-7.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 022f49d hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8088/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8088/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8088/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8088/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8088/console This message was automatically generated.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Quick comments on the patch

        • ContainerStatus is a user-facing record, so you are also sending this to the AMs now. Was that intentional? I was assuming we are decoupling information sent to apps completely from what gets reported from NM to RM.
        • NodeManagers can send the raw information instead of dealing with vcores? Also, send both pmem and vmem usage?
        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Quick comments on the patch ContainerStatus is a user-facing record, so you are also sending this to the AMs now. Was that intentional? I was assuming we are decoupling information sent to apps completely from what gets reported from NM to RM. NodeManagers can send the raw information instead of dealing with vcores? Also, send both pmem and vmem usage?
        Hide
        elgoiri Íñigo Goiri added a comment -

        *I didn't realize about the ContainerStatus being sent to the AM. How would you send the information to the RM then?
        *I can send pmem and vmem with no problems; I'll add it right away. Regarding the CPU utilization, Karthik Kambatla suggested to send millivcores as an int; we would have to agree on what to send there, a float? an int?

        Show
        elgoiri Íñigo Goiri added a comment - *I didn't realize about the ContainerStatus being sent to the AM. How would you send the information to the RM then? *I can send pmem and vmem with no problems; I'll add it right away. Regarding the CPU utilization, Karthik Kambatla suggested to send millivcores as an int; we would have to agree on what to send there, a float? an int?
        Hide
        kasha Karthik Kambatla added a comment -

        For CPU utilization, the other alternative is to send the number of physical cores on the machine along with percentage usage (like the one top shows).

        Vinod Kumar Vavilapalli - does the need to avoid vcores stem from our intention of moving all the utilization information collection to a separate service that might not be specific to Yarn?

        Show
        kasha Karthik Kambatla added a comment - For CPU utilization, the other alternative is to send the number of physical cores on the machine along with percentage usage (like the one top shows). Vinod Kumar Vavilapalli - does the need to avoid vcores stem from our intention of moving all the utilization information collection to a separate service that might not be specific to Yarn?
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        I didn't realize about the ContainerStatus being sent to the AM. How would you send the information to the RM then?

        We will have to create a new record somewhere in the hierarchy of NodeHeartbeatRequest.

        does the need to avoid vcores stem from our intention of moving all the utilization information collection to a separate service that might not be specific to Yarn?

        Only partly. The bigger reason is vcore is more of a construct created for scheduling. We can have a vcore-millis which represents more of the resources reserved as opposed to real utilization. Real utilization can be cpu-wall-time, avg cpu utilization etc.

        For CPU utilization, the other alternative is to send the number of physical cores on the machine along with percentage usage (like the one top shows).

        This reminds me of another thing that I missed mentioning before. We need to send per-container utilization as well as the the host-level utilization.

        May be we should first agree here on what exactly we are sending across. The patch should be relatively straigh-forward after that.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - I didn't realize about the ContainerStatus being sent to the AM. How would you send the information to the RM then? We will have to create a new record somewhere in the hierarchy of NodeHeartbeatRequest. does the need to avoid vcores stem from our intention of moving all the utilization information collection to a separate service that might not be specific to Yarn? Only partly. The bigger reason is vcore is more of a construct created for scheduling. We can have a vcore-millis which represents more of the resources reserved as opposed to real utilization. Real utilization can be cpu-wall-time, avg cpu utilization etc. For CPU utilization, the other alternative is to send the number of physical cores on the machine along with percentage usage (like the one top shows). This reminds me of another thing that I missed mentioning before. We need to send per-container utilization as well as the the host-level utilization. May be we should first agree here on what exactly we are sending across. The patch should be relatively straigh-forward after that.
        Hide
        kasha Karthik Kambatla added a comment -

        The bigger reason is vcore is more of a construct created for scheduling. We can have a vcore-millis which represents more of the resources reserved as opposed to real utilization. Real utilization can be cpu-wall-time, avg cpu utilization etc.

        I see your point. That said, NM is sending information across to RM solely because the scheduler can take it into consideration. It would definitely be simpler if the scheduler doesn't have to convert "real" utilization to vcores. NodeManagers are better places to do that. I would like for the heartbeat itself to have the utilization information in rounded-up vcores on milli-vcores. Should we store both?

        Show
        kasha Karthik Kambatla added a comment - The bigger reason is vcore is more of a construct created for scheduling. We can have a vcore-millis which represents more of the resources reserved as opposed to real utilization. Real utilization can be cpu-wall-time, avg cpu utilization etc. I see your point. That said, NM is sending information across to RM solely because the scheduler can take it into consideration. It would definitely be simpler if the scheduler doesn't have to convert "real" utilization to vcores. NodeManagers are better places to do that. I would like for the heartbeat itself to have the utilization information in rounded-up vcores on milli-vcores. Should we store both?
        Hide
        kasha Karthik Kambatla added a comment -

        I am not sure we want to send per-container utilization. We want to send (1) aggregate usage across all Yarn containers, and the (2) node utilization.

        Show
        kasha Karthik Kambatla added a comment - I am not sure we want to send per-container utilization. We want to send (1) aggregate usage across all Yarn containers, and the (2) node utilization.
        Hide
        elgoiri Íñigo Goiri added a comment -

        For (1), the aggregated usage would be YARN-3481.
        (2) will depend on getting YARN-3534 first; after that, we'll need this one (or YARN-3481) to have the ResourceUtilization entity.

        Show
        elgoiri Íñigo Goiri added a comment - For (1), the aggregated usage would be YARN-3481 . (2) will depend on getting YARN-3534 first; after that, we'll need this one (or YARN-3481 ) to have the ResourceUtilization entity.
        Hide
        elgoiri Íñigo Goiri added a comment -

        Report aggregated utilization.

        Show
        elgoiri Íñigo Goiri added a comment - Report aggregated utilization.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 19m 43s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 7m 43s There were no new javac warning messages.
        +1 javadoc 9m 55s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 2m 31s There were no new checkstyle issues.
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 38s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 5m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api.
        +1 yarn tests 1m 59s Tests passed in hadoop-yarn-common.
        +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common.
        +1 yarn tests 6m 9s Tests passed in hadoop-yarn-server-nodemanager.
            56m 56s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12739650/YARN-1012-8.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 32ffda1
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8253/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8253/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8253/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8253/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8253/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8253/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8253/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 19m 43s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 43s There were no new javac warning messages. +1 javadoc 9m 55s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 31s There were no new checkstyle issues. +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 5m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 59s Tests passed in hadoop-yarn-common. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common. +1 yarn tests 6m 9s Tests passed in hadoop-yarn-server-nodemanager.     56m 56s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739650/YARN-1012-8.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 32ffda1 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8253/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8253/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8253/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8253/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8253/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8253/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8253/console This message was automatically generated.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        I would like for the heartbeat itself to have the utilization information in rounded-up vcores on milli-vcores. Should we store both?

        Sure. Or we can add what is really needed for the immediate scheduling functionality being worked on right now and add more things as we go.

        Did a quick scan of the patch, some top level comments before more detailed review.

        • ResourceUtilizationProto etc are server side records, so we should move them to yarn_server_common_protos.proto
        • NodeHealthStatus is a node's health-report, which is tied to health-checker etc. You can put ResourceUtilization in NodeStatus?
        • ResourceUtilization.setCPUUsage/getCPUUsage -> simply get|setCPU. Usage and Utilization are duplicates.
        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - I would like for the heartbeat itself to have the utilization information in rounded-up vcores on milli-vcores. Should we store both? Sure. Or we can add what is really needed for the immediate scheduling functionality being worked on right now and add more things as we go. Did a quick scan of the patch, some top level comments before more detailed review. ResourceUtilizationProto etc are server side records, so we should move them to yarn_server_common_protos.proto NodeHealthStatus is a node's health-report, which is tied to health-checker etc. You can put ResourceUtilization in NodeStatus? ResourceUtilization.setCPUUsage/getCPUUsage -> simply get|setCPU. Usage and Utilization are duplicates.
        Hide
        kasha Karthik Kambatla added a comment -

        Sorry for the delay in getting around to this. In addition to Vinod's comments, a few more:

        1. Unused imports added in ContainerImpl, MockContainer, NodeHealthStatusPBImpl, NodeStatusUpdaterImpl
        2. ContainersMontiorImpl
          1. Rename auxContainersUtilization to trackedContainersUtilization or tmpContainersUtilization for clarity. auxContainersUtilization could lead people to think about auxiliary services.
          2. Nit: might be premature optimization. Instead of ResourceUtilizations#addTo(ResourceUtilization), did we consider ResourceUtilization#addTo(pmem, vmem, cpu)?
        3. NodeStatusUpdaterImpl: Instead of adding a getContainersUtilization method and using the returned value immediately, how about adding a setContainersUtilizationInHealthStatus?
        4. ResourceUtilization
          1. License header has some unrelated numbers
          2. In hashCode, instead of using 939769357, we should probably do prime * result?
        5. ResourceUtilizations
          1. Looks like we only use addTo, how about getting rid of the remaining methods for now?
          2. Also, if we move that addTo to ResourceUtilization itself, we will not be needing another class.
          3. If we do choose to keep the class, we should probably annotate it Private and not LimitedPrivate. Do we see MapReduce using this?
        Show
        kasha Karthik Kambatla added a comment - Sorry for the delay in getting around to this. In addition to Vinod's comments, a few more: Unused imports added in ContainerImpl, MockContainer, NodeHealthStatusPBImpl, NodeStatusUpdaterImpl ContainersMontiorImpl Rename auxContainersUtilization to trackedContainersUtilization or tmpContainersUtilization for clarity. auxContainersUtilization could lead people to think about auxiliary services. Nit: might be premature optimization. Instead of ResourceUtilizations#addTo(ResourceUtilization) , did we consider ResourceUtilization#addTo(pmem, vmem, cpu) ? NodeStatusUpdaterImpl: Instead of adding a getContainersUtilization method and using the returned value immediately, how about adding a setContainersUtilizationInHealthStatus? ResourceUtilization License header has some unrelated numbers In hashCode , instead of using 939769357, we should probably do prime * result ? ResourceUtilizations Looks like we only use addTo, how about getting rid of the remaining methods for now? Also, if we move that addTo to ResourceUtilization itself, we will not be needing another class. If we do choose to keep the class, we should probably annotate it Private and not LimitedPrivate. Do we see MapReduce using this?
        Hide
        elgoiri Íñigo Goiri added a comment -

        Fixed based on feedback from Vinod and Karthik. The most important change is moving ResourceUtilization to the server part. Still not sure about the audience and stability stuff being correct.

        Show
        elgoiri Íñigo Goiri added a comment - Fixed based on feedback from Vinod and Karthik. The most important change is moving ResourceUtilization to the server part. Still not sure about the audience and stability stuff being correct.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 17m 32s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 36s There were no new javac warning messages.
        +1 javadoc 9m 36s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 15s The applied patch generated 8 new checkstyle issues (total was 19, now 27).
        -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 37s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        -1 findbugs 3m 11s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 6m 55s Tests passed in hadoop-yarn-client.
        +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common.
        +1 yarn tests 6m 6s Tests passed in hadoop-yarn-server-nodemanager.
            55m 22s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-common



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12743609/YARN-1012-9.patch
        Optional Tests javac unit findbugs checkstyle javadoc
        git revision trunk / 2eae130
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8427/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8427/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8427/artifact/patchprocess/whitespace.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8427/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
        hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8427/artifact/patchprocess/testrun_hadoop-yarn-client.txt
        hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8427/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8427/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8427/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8427/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 32s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 15s The applied patch generated 8 new checkstyle issues (total was 19, now 27). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 3m 11s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings. +1 yarn tests 6m 55s Tests passed in hadoop-yarn-client. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common. +1 yarn tests 6m 6s Tests passed in hadoop-yarn-server-nodemanager.     55m 22s   Reason Tests FindBugs module:hadoop-yarn-server-common Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12743609/YARN-1012-9.patch Optional Tests javac unit findbugs checkstyle javadoc git revision trunk / 2eae130 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8427/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8427/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8427/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8427/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8427/artifact/patchprocess/testrun_hadoop-yarn-client.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8427/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8427/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8427/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8427/console This message was automatically generated.
        Hide
        kasha Karthik Kambatla added a comment -

        Thanks for updating the patch, Inigo. Looks better, few minor comments:

        1. ResourceUtilization of containers is calculated only when container monitoring is enabled. Given the low overhead of sending this across, I think it is okay to not add another config to choose whether to send this information across or not. However, if container-monitoring is not enabled, we should probably not send it across. Accordingly, would it make sense to not modify NodeStatus#newInstance and just call the setter when required.
        2. ResourceUtilization
          1. javadoc has an error
             *
             * @see Resource
             */
            
          2. Now that we have moved it to be server-side only, does it need to Public? How about changing it to Private-Evolving
          3. We don't need to add annotations for individual methods if it is the same as the enclosing class.
        3. ContainersMonitorImpl - Extra space below
                   // Save the aggregated utilization of the containers
                  setContainersUtilization(trackedContainersUtilization );
          
        4. NodeStatus doesn't need to ResourceUtilization
        5. NodeStatusUpdaterImpl doesn't need to import Records
        Show
        kasha Karthik Kambatla added a comment - Thanks for updating the patch, Inigo. Looks better, few minor comments: ResourceUtilization of containers is calculated only when container monitoring is enabled. Given the low overhead of sending this across, I think it is okay to not add another config to choose whether to send this information across or not. However, if container-monitoring is not enabled, we should probably not send it across. Accordingly, would it make sense to not modify NodeStatus#newInstance and just call the setter when required. ResourceUtilization javadoc has an error * * @see Resource */ Now that we have moved it to be server-side only, does it need to Public? How about changing it to Private-Evolving We don't need to add annotations for individual methods if it is the same as the enclosing class. ContainersMonitorImpl - Extra space below // Save the aggregated utilization of the containers setContainersUtilization(trackedContainersUtilization ); NodeStatus doesn't need to ResourceUtilization NodeStatusUpdaterImpl doesn't need to import Records
        Hide
        elgoiri Íñigo Goiri added a comment -

        Thanks for the feedback, it makes sense.
        I have one doubt though; for container monitoring, are you referring to YarnConfiguration.NM_CONTAINER_METRICS_ENABLE? You want to do the following?

            if (this.containerMetricsEnabled) {
              nodeStatus.setContainersUtilization(containersUtilization);
            }
        
        Show
        elgoiri Íñigo Goiri added a comment - Thanks for the feedback, it makes sense. I have one doubt though; for container monitoring, are you referring to YarnConfiguration.NM_CONTAINER_METRICS_ENABLE? You want to do the following? if ( this .containerMetricsEnabled) { nodeStatus.setContainersUtilization(containersUtilization); }
        Hide
        kasha Karthik Kambatla added a comment -

        Actually, looks like we always monitor the utilization, but enforce it only when

        {p|v}

        mem-check-enabled. So, I guess we are good.

        Show
        kasha Karthik Kambatla added a comment - Actually, looks like we always monitor the utilization, but enforce it only when {p|v} mem-check-enabled. So, I guess we are good.
        Hide
        elgoiri Íñigo Goiri added a comment -

        Improved patch based on feedback.

        Show
        elgoiri Íñigo Goiri added a comment - Improved patch based on feedback.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 17m 37s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 36s There were no new javac warning messages.
        +1 javadoc 9m 37s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 10s The applied patch generated 2 new checkstyle issues (total was 19, now 21).
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        -1 findbugs 3m 11s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 6m 52s Tests passed in hadoop-yarn-client.
        +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common.
        +1 yarn tests 6m 14s Tests passed in hadoop-yarn-server-nodemanager.
            55m 27s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-common



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12743646/YARN-1012-10.patch
        Optional Tests javac unit findbugs checkstyle javadoc
        git revision trunk / 688617d
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8430/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8430/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8430/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
        hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8430/artifact/patchprocess/testrun_hadoop-yarn-client.txt
        hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8430/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8430/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8430/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8430/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 37s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 10s The applied patch generated 2 new checkstyle issues (total was 19, now 21). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 3m 11s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings. +1 yarn tests 6m 52s Tests passed in hadoop-yarn-client. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common. +1 yarn tests 6m 14s Tests passed in hadoop-yarn-server-nodemanager.     55m 27s   Reason Tests FindBugs module:hadoop-yarn-server-common Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12743646/YARN-1012-10.patch Optional Tests javac unit findbugs checkstyle javadoc git revision trunk / 688617d Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8430/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8430/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8430/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8430/artifact/patchprocess/testrun_hadoop-yarn-client.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8430/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8430/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8430/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8430/console This message was automatically generated.
        Hide
        kasha Karthik Kambatla added a comment -

        Íñigo Goiri - could you look into the checkstyle and findbugs warnings please?

        Show
        kasha Karthik Kambatla added a comment - Íñigo Goiri - could you look into the checkstyle and findbugs warnings please?
        Hide
        elgoiri Íñigo Goiri added a comment -

        Fixed checkstyle issues (I hope the package-info is done properly).
        I could not find a reason for the FindBug; in another patch it just disappeared so let's hope this is the case.

        Show
        elgoiri Íñigo Goiri added a comment - Fixed checkstyle issues (I hope the package-info is done properly). I could not find a reason for the FindBug; in another patch it just disappeared so let's hope this is the case.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 17m 39s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 52s There were no new javac warning messages.
        +1 javadoc 9m 44s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 23s There were no new checkstyle issues.
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 23s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        -1 findbugs 3m 15s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 7m 1s Tests passed in hadoop-yarn-client.
        +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common.
        -1 yarn tests 6m 4s Tests failed in hadoop-yarn-server-nodemanager.
            55m 45s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-common
        Failed unit tests hadoop.yarn.server.nodemanager.containermanager.linux.privileged.TestPrivilegedOperationExecutor



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12744329/YARN-1012-11.patch
        Optional Tests javac unit findbugs checkstyle javadoc
        git revision trunk / 2e3d83f
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8459/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8459/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
        hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8459/artifact/patchprocess/testrun_hadoop-yarn-client.txt
        hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8459/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8459/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8459/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8459/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 39s Pre-patch trunk has 3 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 52s There were no new javac warning messages. +1 javadoc 9m 44s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 23s There were no new checkstyle issues. +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 23s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 3m 15s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings. +1 yarn tests 7m 1s Tests passed in hadoop-yarn-client. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common. -1 yarn tests 6m 4s Tests failed in hadoop-yarn-server-nodemanager.     55m 45s   Reason Tests FindBugs module:hadoop-yarn-server-common Failed unit tests hadoop.yarn.server.nodemanager.containermanager.linux.privileged.TestPrivilegedOperationExecutor Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744329/YARN-1012-11.patch Optional Tests javac unit findbugs checkstyle javadoc git revision trunk / 2e3d83f Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8459/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8459/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html hadoop-yarn-client test log https://builds.apache.org/job/PreCommit-YARN-Build/8459/artifact/patchprocess/testrun_hadoop-yarn-client.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8459/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8459/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8459/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8459/console This message was automatically generated.
        Hide
        kasha Karthik Kambatla added a comment -

        Both the test result and findbugs warnings look unrelated.

        +1. Checking this in.

        Show
        kasha Karthik Kambatla added a comment - Both the test result and findbugs warnings look unrelated. +1. Checking this in.
        Hide
        elgoiri Íñigo Goiri added a comment -

        Thank you Karthik Kambatla!
        Once commited, I'm moving to YARN-3534 to reuse the ResourceUtilization.

        Show
        elgoiri Íñigo Goiri added a comment - Thank you Karthik Kambatla ! Once commited, I'm moving to YARN-3534 to reuse the ResourceUtilization.
        Hide
        kasha Karthik Kambatla added a comment -

        Just committed this to trunk and branch-2.

        Thanks a bunch for working on this critical piece and your patience with the reviews, Inigo.

        Show
        kasha Karthik Kambatla added a comment - Just committed this to trunk and branch-2. Thanks a bunch for working on this critical piece and your patience with the reviews, Inigo.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8140 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8140/)
        YARN-1012. Report NM aggregated container resource utilization in heartbeat. (Inigo Goiri via kasha) (kasha: rev 527c40e4d664c721b8f32d7cd8df21b2666fea8a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/ResourceUtilizationPBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/package-info.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitor.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/ResourceUtilization.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8140 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8140/ ) YARN-1012 . Report NM aggregated container resource utilization in heartbeat. (Inigo Goiri via kasha) (kasha: rev 527c40e4d664c721b8f32d7cd8df21b2666fea8a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/ResourceUtilizationPBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/package-info.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/ResourceUtilization.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #252 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/252/)
        YARN-1012. Report NM aggregated container resource utilization in heartbeat. (Inigo Goiri via kasha) (kasha: rev 527c40e4d664c721b8f32d7cd8df21b2666fea8a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/package-info.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/ResourceUtilizationPBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/ResourceUtilization.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitor.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #252 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/252/ ) YARN-1012 . Report NM aggregated container resource utilization in heartbeat. (Inigo Goiri via kasha) (kasha: rev 527c40e4d664c721b8f32d7cd8df21b2666fea8a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/package-info.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/ResourceUtilizationPBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/ResourceUtilization.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #982 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/982/)
        YARN-1012. Report NM aggregated container resource utilization in heartbeat. (Inigo Goiri via kasha) (kasha: rev 527c40e4d664c721b8f32d7cd8df21b2666fea8a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/ResourceUtilization.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/package-info.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitor.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/ResourceUtilizationPBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #982 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/982/ ) YARN-1012 . Report NM aggregated container resource utilization in heartbeat. (Inigo Goiri via kasha) (kasha: rev 527c40e4d664c721b8f32d7cd8df21b2666fea8a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/ResourceUtilization.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/package-info.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/ResourceUtilizationPBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2179 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2179/)
        YARN-1012. Report NM aggregated container resource utilization in heartbeat. (Inigo Goiri via kasha) (kasha: rev 527c40e4d664c721b8f32d7cd8df21b2666fea8a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/package-info.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/ResourceUtilization.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/ResourceUtilizationPBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitor.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2179 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2179/ ) YARN-1012 . Report NM aggregated container resource utilization in heartbeat. (Inigo Goiri via kasha) (kasha: rev 527c40e4d664c721b8f32d7cd8df21b2666fea8a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/package-info.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/ResourceUtilization.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/ResourceUtilizationPBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitor.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2198 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2198/)
        YARN-1012. Report NM aggregated container resource utilization in heartbeat. (Inigo Goiri via kasha) (kasha: rev 527c40e4d664c721b8f32d7cd8df21b2666fea8a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitor.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/ResourceUtilizationPBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/ResourceUtilization.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/package-info.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2198 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2198/ ) YARN-1012 . Report NM aggregated container resource utilization in heartbeat. (Inigo Goiri via kasha) (kasha: rev 527c40e4d664c721b8f32d7cd8df21b2666fea8a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/ResourceUtilizationPBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/ResourceUtilization.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/package-info.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #240 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/240/)
        YARN-1012. Report NM aggregated container resource utilization in heartbeat. (Inigo Goiri via kasha) (kasha: rev 527c40e4d664c721b8f32d7cd8df21b2666fea8a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/package-info.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/ResourceUtilization.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/ResourceUtilizationPBImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitor.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #240 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/240/ ) YARN-1012 . Report NM aggregated container resource utilization in heartbeat. (Inigo Goiri via kasha) (kasha: rev 527c40e4d664c721b8f32d7cd8df21b2666fea8a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/package-info.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/ResourceUtilization.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/ResourceUtilizationPBImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitor.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #250 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/250/)
        YARN-1012. Report NM aggregated container resource utilization in heartbeat. (Inigo Goiri via kasha) (kasha: rev 527c40e4d664c721b8f32d7cd8df21b2666fea8a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/package-info.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/ResourceUtilization.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/ResourceUtilizationPBImpl.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitor.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #250 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/250/ ) YARN-1012 . Report NM aggregated container resource utilization in heartbeat. (Inigo Goiri via kasha) (kasha: rev 527c40e4d664c721b8f32d7cd8df21b2666fea8a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/package-info.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/NodeStatus.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/ResourceUtilization.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/ResourceUtilizationPBImpl.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/impl/pb/NodeStatusPBImpl.java
        Hide
        srikanthkandula Srikanth Kandula added a comment -

        Íñigo Goiri, Karthik Kambatla Could you comment on whether this should go into hadoop common. Also, it may be worthwhile to extend this to also account for network and disk usages of the containers... See Hadoop 12210.

        Show
        srikanthkandula Srikanth Kandula added a comment - Íñigo Goiri , Karthik Kambatla Could you comment on whether this should go into hadoop common. Also, it may be worthwhile to extend this to also account for network and disk usages of the containers... See Hadoop 12210.
        Hide
        elgoiri Íñigo Goiri added a comment -

        I think this is very YARN specific. It relies on the ResourceCalculator and so on which come from Common though.

        Regarding adding network and disk usage, I fully agree. You guys should first extend ResourceUtilization (as done in this patch) to support disk and network and then extend the node resource monitor (YARN-3534) to collect it from the node.

        Show
        elgoiri Íñigo Goiri added a comment - I think this is very YARN specific. It relies on the ResourceCalculator and so on which come from Common though. Regarding adding network and disk usage, I fully agree. You guys should first extend ResourceUtilization (as done in this patch) to support disk and network and then extend the node resource monitor ( YARN-3534 ) to collect it from the node.
        Hide
        srikanthkandula Srikanth Kandula added a comment -

        Ack. Will do.

        Show
        srikanthkandula Srikanth Kandula added a comment - Ack. Will do.

          People

          • Assignee:
            elgoiri Íñigo Goiri
            Reporter:
            acmurthy Arun C Murthy
          • Votes:
            0 Vote for this issue
            Watchers:
            20 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development