Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3978

Configurably turn off the saving of container info in Generic AHS

    Details

      Description

      Depending on how each application's metadata is stored, one week's worth of data stored in the Generic Application History Server's database can grow to be almost a terabyte of local disk space. In order to alleviate this, I suggest that there is a need for a configuration option to turn off saving of non-AM container metadata in the GAHS data store.

      1. YARN-3978.001.patch
        11 kB
        Eric Payne
      2. YARN-3978.002.patch
        18 kB
        Eric Payne
      3. YARN-3978.003.patch
        19 kB
        Eric Payne
      4. YARN-3978.004.patch
        19 kB
        Eric Payne

        Issue Links

          Activity

          Hide
          eepayne Eric Payne added a comment -

          Use Case: A user launches an application on a secured cluster that runs for some time and then fails within the AM (perhaps due to OOM in the AM), leaving no history in the job history server. The user doesn't notice that the job has failed until after the application has dropped off of the RM's application store. At this point, if no information was stored in the Generic Application History Service, a user must rely on a priviledged system administrator to access the AM logs for them.

          It is desirable to activate the Generic Application History service within the timeline server so that users can access their application's information even after the RM has forgotten about their application. This app information should be kept in the GAHS for 1 week, as is done, for example, for logs in the job history server.

          One way that the Generic AHS stores metadata about an application is in an Entity levelDB. This includes information about each container for each application. Based on my analysis, the levelDB size grows by at least 2500 bytes per container (uncompressed). This is a conservative estimate as the size could be much bigger based on the amount of diagnostic information associated with failed containers.

          On very large and busy clusters, the amount needed on the timeline server's local disk would be between 0.6 TB and 1.0 TB (uncompressed). Even if we assume 90% compression, that's still between 60 GB and 100 GB that will be needed on the local disk. In addition to this, between 80 GB and 143 GB of metadata (uncopressed) will need to be cleaned up every day from the levelDB, which will delay other processing in the timeline server.

          The proposal of this JIRA is to add a configuration property that enables/disables whether or not the GAHS stores container information in the levelDB. Whith this change, I estimate that the local disk usage would be about 5700 bytes per job, or about 10 GB (uncompressed) per week. Additionally, the daily cleanup load would only be about 1.5 GB per day.

          Show
          eepayne Eric Payne added a comment - Use Case: A user launches an application on a secured cluster that runs for some time and then fails within the AM (perhaps due to OOM in the AM), leaving no history in the job history server. The user doesn't notice that the job has failed until after the application has dropped off of the RM's application store. At this point, if no information was stored in the Generic Application History Service, a user must rely on a priviledged system administrator to access the AM logs for them. It is desirable to activate the Generic Application History service within the timeline server so that users can access their application's information even after the RM has forgotten about their application. This app information should be kept in the GAHS for 1 week, as is done, for example, for logs in the job history server. One way that the Generic AHS stores metadata about an application is in an Entity levelDB. This includes information about each container for each application. Based on my analysis, the levelDB size grows by at least 2500 bytes per container (uncompressed). This is a conservative estimate as the size could be much bigger based on the amount of diagnostic information associated with failed containers. On very large and busy clusters, the amount needed on the timeline server's local disk would be between 0.6 TB and 1.0 TB (uncompressed). Even if we assume 90% compression, that's still between 60 GB and 100 GB that will be needed on the local disk. In addition to this, between 80 GB and 143 GB of metadata (uncopressed) will need to be cleaned up every day from the levelDB, which will delay other processing in the timeline server. The proposal of this JIRA is to add a configuration property that enables/disables whether or not the GAHS stores container information in the levelDB. Whith this change, I estimate that the local disk usage would be about 5700 bytes per job, or about 10 GB (uncompressed) per week. Additionally, the daily cleanup load would only be about 1.5 GB per day.
          Hide
          eepayne Eric Payne added a comment -

          Attaching version 001 of the patch. @jeagles, would you like to take a look?

          Show
          eepayne Eric Payne added a comment - Attaching version 001 of the patch. @jeagles, would you like to take a look?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 18m 41s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 50s There were no new javac warning messages.
          +1 javadoc 9m 43s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 48s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          -1 checkstyle 2m 4s The applied patch generated 2 new checkstyle issues (total was 39, now 41).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 23s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          -1 findbugs 4m 6s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common.
          -1 yarn tests 52m 35s Tests failed in hadoop-yarn-server-resourcemanager.
              98m 24s  



          Reason Tests
          FindBugs module:hadoop-yarn-server-resourcemanager
          Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestChildQueueOrder
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations
            hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
            hadoop.yarn.server.resourcemanager.TestClientRMService



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12747458/YARN-3978.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 3e6fce9
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8688/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8688/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8688/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8688/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8688/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8688/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8688/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8688/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8688/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 41s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 50s There were no new javac warning messages. +1 javadoc 9m 43s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 48s The applied patch generated 1 new checkstyle issues (total was 211, now 211). -1 checkstyle 2m 4s The applied patch generated 2 new checkstyle issues (total was 39, now 41). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 23s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 4m 6s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common. -1 yarn tests 52m 35s Tests failed in hadoop-yarn-server-resourcemanager.     98m 24s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestChildQueueOrder   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations   hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler   hadoop.yarn.server.resourcemanager.TestClientRMService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12747458/YARN-3978.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3e6fce9 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8688/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8688/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8688/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8688/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8688/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8688/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8688/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8688/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8688/console This message was automatically generated.
          Hide
          eepayne Eric Payne added a comment -

          Attaching version 002 of the patch. Should address most of the issues. After the next build report, I will comment on the ones I intentionally am not fixing (e.g., checkstyle warnings about file length).

          Show
          eepayne Eric Payne added a comment - Attaching version 002 of the patch. Should address most of the issues. After the next build report, I will comment on the ones I intentionally am not fixing (e.g., checkstyle warnings about file length).
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 18m 48s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 9m 21s There were no new javac warning messages.
          +1 javadoc 12m 2s There were no new javadoc warning messages.
          +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse.
          -1 findbugs 4m 58s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 31s Tests passed in hadoop-yarn-api.
          +1 yarn tests 0m 29s Tests passed in hadoop-yarn-server-common.
          -1 yarn tests 54m 33s Tests failed in hadoop-yarn-server-resourcemanager.
              105m 54s  



          Reason Tests
          FindBugs module:hadoop-yarn-server-resourcemanager
          Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12747634/YARN-3978.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c020b62
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8707/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8707/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8707/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8707/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8707/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8707/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8707/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8707/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 48s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 9m 21s There were no new javac warning messages. +1 javadoc 12m 2s There were no new javadoc warning messages. +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 36s The patch built with eclipse:eclipse. -1 findbugs 4m 58s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 31s Tests passed in hadoop-yarn-api. +1 yarn tests 0m 29s Tests passed in hadoop-yarn-server-common. -1 yarn tests 54m 33s Tests failed in hadoop-yarn-server-resourcemanager.     105m 54s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12747634/YARN-3978.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c020b62 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8707/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8707/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8707/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8707/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8707/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8707/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8707/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8707/console This message was automatically generated.
          Hide
          eepayne Eric Payne added a comment -

          Version 003 of patch.

          Show
          eepayne Eric Payne added a comment - Version 003 of patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 18m 14s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 7m 43s There were no new javac warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 41s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 19s mvn install still works.
          +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
          +1 findbugs 3m 57s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 21s Tests passed in hadoop-yarn-api.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 52m 15s Tests passed in hadoop-yarn-server-resourcemanager.
              97m 1s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12748033/YARN-3978.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 91b42e7
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8719/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8719/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8719/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8719/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8719/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8719/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8719/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 14s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 7m 43s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 41s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 19s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 3m 57s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 21s Tests passed in hadoop-yarn-api. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common. +1 yarn tests 52m 15s Tests passed in hadoop-yarn-server-resourcemanager.     97m 1s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748033/YARN-3978.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 91b42e7 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8719/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8719/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8719/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8719/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8719/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8719/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8719/console This message was automatically generated.
          Hide
          eepayne Eric Payne added a comment -

          checkstyle indicates that YarnConfiguration.java is too long. I will not be fixing that as part of this JIRA. Everything else from the build seems to be okay.

          Jonathan Eagles, can you please have a look at this patch?

          Show
          eepayne Eric Payne added a comment - checkstyle indicates that YarnConfiguration.java is too long. I will not be fixing that as part of this JIRA. Everything else from the build seems to be okay. Jonathan Eagles , can you please have a look at this patch?
          Hide
          jeagles Jonathan Eagles added a comment -

          Eric Payne, this patch looks mainly good. This feature will be good going forward for those who want a scalable generic history solution until timeline server v2 is available. Couple of minor things.

          • It is not evident to me the changes made to TestClientRMService, TestChildQueueOrder, TestLeafQueue, TestReservations, and TestFifoScheduler. Could you help to explain those changes or remove them if they are extra?
          • Please update the comment

            + // Store system metrics for all containers only when storeContainerMetaInfo
            + // is true.

            To indicate that AM metrics publishing are delayed until later in this scenario.

          • Is there a better configuration name that could be used? save-container-meta-info doesn't convey that AM container info is still published if this flag is disabled.
          Show
          jeagles Jonathan Eagles added a comment - Eric Payne , this patch looks mainly good. This feature will be good going forward for those who want a scalable generic history solution until timeline server v2 is available. Couple of minor things. It is not evident to me the changes made to TestClientRMService, TestChildQueueOrder, TestLeafQueue, TestReservations, and TestFifoScheduler. Could you help to explain those changes or remove them if they are extra? Please update the comment + // Store system metrics for all containers only when storeContainerMetaInfo + // is true. To indicate that AM metrics publishing are delayed until later in this scenario. Is there a better configuration name that could be used? save-container-meta-info doesn't convey that AM container info is still published if this flag is disabled.
          Hide
          eepayne Eric Payne added a comment -

          Thanks very much Jonathan Eagles for your review and comments.

          It is not evident to me the changes made to TestClientRMService, TestChildQueueOrder, TestLeafQueue, TestReservations, and TestFifoScheduler. Could you help to explain those changes or remove them if they are extra?

          In each of these tests, the rmcontext is mocked before constructing a new instance of RMContainerImpl, but the getYarnConfiguration method is not handled. Since this patch adds a dependency on rmContext.getYarnConfiguration() in the constructor for RMContainerImpl, an explicit mock for getYarnConfiguration had to be added in these tests to prevent NPE.

          Please update the comment
          + // Store system metrics for all containers only when storeContainerMetaInfo
          + // is true.
          To indicate that AM metrics publishing are delayed until later in this scenario.

          Done

          Is there a better configuration name that could be used? save-container-meta-info doesn't convey that AM container info is still published if this flag is disabled.

          How about save-non-am-container-meta-info? I thought about save-only-am-container-meta-info, but then true would mean that publishing of non-am containers would be turned off, and I thought that was too confusing.

          Show
          eepayne Eric Payne added a comment - Thanks very much Jonathan Eagles for your review and comments. It is not evident to me the changes made to TestClientRMService, TestChildQueueOrder, TestLeafQueue, TestReservations, and TestFifoScheduler. Could you help to explain those changes or remove them if they are extra? In each of these tests, the rmcontext is mocked before constructing a new instance of RMContainerImpl , but the getYarnConfiguration method is not handled. Since this patch adds a dependency on rmContext.getYarnConfiguration() in the constructor for RMContainerImpl , an explicit mock for getYarnConfiguration had to be added in these tests to prevent NPE. Please update the comment + // Store system metrics for all containers only when storeContainerMetaInfo + // is true. To indicate that AM metrics publishing are delayed until later in this scenario. Done Is there a better configuration name that could be used? save-container-meta-info doesn't convey that AM container info is still published if this flag is disabled. How about save-non-am-container-meta-info ? I thought about save-only-am-container-meta-info , but then true would mean that publishing of non-am containers would be turned off, and I thought that was too confusing.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 18m 45s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 7m 43s There were no new javac warning messages.
          +1 javadoc 9m 46s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 44s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 25s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 4m 4s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 52m 54s Tests passed in hadoop-yarn-server-resourcemanager.
              98m 37s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12748277/YARN-3978.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d311a38
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8748/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8748/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8748/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8748/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8748/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8748/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8748/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 45s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 7m 43s There were no new javac warning messages. +1 javadoc 9m 46s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 44s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 25s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 4m 4s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common. +1 yarn tests 52m 54s Tests passed in hadoop-yarn-server-resourcemanager.     98m 37s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748277/YARN-3978.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d311a38 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8748/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8748/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8748/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8748/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8748/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8748/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8748/console This message was automatically generated.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8255 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8255/)
          YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8255 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8255/ ) YARN-3978 . Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
          Hide
          jeagles Jonathan Eagles added a comment -

          +1. Great addition, Eric Payne

          Show
          jeagles Jonathan Eagles added a comment - +1. Great addition, Eric Payne
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #277 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/277/)
          YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #277 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/277/ ) YARN-3978 . Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1007 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1007/)
          YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1007 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1007/ ) YARN-3978 . Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2204 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2204/)
          YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2204 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2204/ ) YARN-3978 . Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2223 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2223/)
          YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2223 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2223/ ) YARN-3978 . Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #266 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/266/)
          YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #266 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/266/ ) YARN-3978 . Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #274 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/274/)
          YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #274 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/274/ ) YARN-3978 . Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Pulled this into 2.6.1. The patch applies cleanly for the most part except for a couple of minor merge conflicts in test-cases which I fixed.

          Ran compilation and TestClientRMService, TestRMContainerImpl, TestChildQueueOrder, TestLeafQueue, TestReservations, TestFifoScheduler before the push.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Pulled this into 2.6.1. The patch applies cleanly for the most part except for a couple of minor merge conflicts in test-cases which I fixed. Ran compilation and TestClientRMService, TestRMContainerImpl, TestChildQueueOrder, TestLeafQueue, TestReservations, TestFifoScheduler before the push.

            People

            • Assignee:
              eepayne Eric Payne
              Reporter:
              eepayne Eric Payne
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development