Details
-
Type:
Improvement
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 2.8.0, 2.7.1
-
Fix Version/s: 2.6.1, 2.8.0, 2.7.2, 3.0.0-alpha1
-
Component/s: timelineserver, yarn
-
Labels:
Description
Depending on how each application's metadata is stored, one week's worth of data stored in the Generic Application History Server's database can grow to be almost a terabyte of local disk space. In order to alleviate this, I suggest that there is a need for a configuration option to turn off saving of non-AM container metadata in the GAHS data store.
Issue Links
- is related to
-
YARN-4688
Allow YARN system metrics publisher to use ATS v1.5 APIs
-
- Open
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Attaching version 001 of the patch. @jeagles, would you like to take a look?
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| -1 | pre-patch | 18m 41s | Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | javac | 7m 50s | There were no new javac warning messages. |
| +1 | javadoc | 9m 43s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
| -1 | checkstyle | 1m 48s | The applied patch generated 1 new checkstyle issues (total was 211, now 211). |
| -1 | checkstyle | 2m 4s | The applied patch generated 2 new checkstyle issues (total was 39, now 41). |
| +1 | whitespace | 0m 1s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 23s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
| -1 | findbugs | 4m 6s | The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. |
| +1 | yarn tests | 0m 24s | Tests passed in hadoop-yarn-api. |
| +1 | yarn tests | 0m 25s | Tests passed in hadoop-yarn-server-common. |
| -1 | yarn tests | 52m 35s | Tests failed in hadoop-yarn-server-resourcemanager. |
| 98m 24s |
| Reason | Tests |
|---|---|
| FindBugs | module:hadoop-yarn-server-resourcemanager |
| Failed unit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestChildQueueOrder |
| hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue | |
| hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations | |
| hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler | |
| hadoop.yarn.server.resourcemanager.TestClientRMService |
This message was automatically generated.
Attaching version 002 of the patch. Should address most of the issues. After the next build report, I will comment on the ones I intentionally am not fixing (e.g., checkstyle warnings about file length).
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| -1 | pre-patch | 18m 48s | Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 6 new or modified test files. |
| +1 | javac | 9m 21s | There were no new javac warning messages. |
| +1 | javadoc | 12m 2s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 26s | The applied patch does not increase the total number of release audit warnings. |
| -1 | checkstyle | 1m 51s | The applied patch generated 1 new checkstyle issues (total was 211, now 211). |
| +1 | whitespace | 0m 2s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 37s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 36s | The patch built with eclipse:eclipse. |
| -1 | findbugs | 4m 58s | The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. |
| +1 | yarn tests | 0m 31s | Tests passed in hadoop-yarn-api. |
| +1 | yarn tests | 0m 29s | Tests passed in hadoop-yarn-server-common. |
| -1 | yarn tests | 54m 33s | Tests failed in hadoop-yarn-server-resourcemanager. |
| 105m 54s |
| Reason | Tests |
|---|---|
| FindBugs | module:hadoop-yarn-server-resourcemanager |
| Failed unit tests | hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler |
| hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority |
This message was automatically generated.
Version 003 of patch.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| -1 | pre-patch | 18m 14s | Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 6 new or modified test files. |
| +1 | javac | 7m 43s | There were no new javac warning messages. |
| +1 | javadoc | 9m 40s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 21s | The applied patch does not increase the total number of release audit warnings. |
| -1 | checkstyle | 1m 41s | The applied patch generated 1 new checkstyle issues (total was 211, now 211). |
| +1 | whitespace | 0m 2s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 19s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 31s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 3m 57s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| +1 | yarn tests | 0m 21s | Tests passed in hadoop-yarn-api. |
| +1 | yarn tests | 0m 24s | Tests passed in hadoop-yarn-server-common. |
| +1 | yarn tests | 52m 15s | Tests passed in hadoop-yarn-server-resourcemanager. |
| 97m 1s |
This message was automatically generated.
checkstyle indicates that YarnConfiguration.java is too long. I will not be fixing that as part of this JIRA. Everything else from the build seems to be okay.
Jonathan Eagles, can you please have a look at this patch?
Eric Payne, this patch looks mainly good. This feature will be good going forward for those who want a scalable generic history solution until timeline server v2 is available. Couple of minor things.
- It is not evident to me the changes made to TestClientRMService, TestChildQueueOrder, TestLeafQueue, TestReservations, and TestFifoScheduler. Could you help to explain those changes or remove them if they are extra?
- Please update the comment
+ // Store system metrics for all containers only when storeContainerMetaInfo
+ // is true.To indicate that AM metrics publishing are delayed until later in this scenario.
- Is there a better configuration name that could be used? save-container-meta-info doesn't convey that AM container info is still published if this flag is disabled.
Thanks very much Jonathan Eagles for your review and comments.
It is not evident to me the changes made to TestClientRMService, TestChildQueueOrder, TestLeafQueue, TestReservations, and TestFifoScheduler. Could you help to explain those changes or remove them if they are extra?
In each of these tests, the rmcontext is mocked before constructing a new instance of RMContainerImpl, but the getYarnConfiguration method is not handled. Since this patch adds a dependency on rmContext.getYarnConfiguration() in the constructor for RMContainerImpl, an explicit mock for getYarnConfiguration had to be added in these tests to prevent NPE.
Please update the comment
+ // Store system metrics for all containers only when storeContainerMetaInfo
+ // is true.
To indicate that AM metrics publishing are delayed until later in this scenario.
Done
Is there a better configuration name that could be used? save-container-meta-info doesn't convey that AM container info is still published if this flag is disabled.
How about save-non-am-container-meta-info? I thought about save-only-am-container-meta-info, but then true would mean that publishing of non-am containers would be turned off, and I thought that was too confusing.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| -1 | pre-patch | 18m 45s | Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | tests included | 0m 0s | The patch appears to include 6 new or modified test files. |
| +1 | javac | 7m 43s | There were no new javac warning messages. |
| +1 | javadoc | 9m 46s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
| -1 | checkstyle | 1m 44s | The applied patch generated 1 new checkstyle issues (total was 211, now 211). |
| +1 | whitespace | 0m 1s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 25s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 4m 4s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| +1 | yarn tests | 0m 23s | Tests passed in hadoop-yarn-api. |
| +1 | yarn tests | 0m 25s | Tests passed in hadoop-yarn-server-common. |
| +1 | yarn tests | 52m 54s | Tests passed in hadoop-yarn-server-resourcemanager. |
| 98m 37s |
This message was automatically generated.
FAILURE: Integrated in Hadoop-trunk-Commit #8255 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8255/)
YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
+1. Great addition, Eric Payne
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #277 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/277/)
YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
FAILURE: Integrated in Hadoop-Yarn-trunk #1007 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1007/)
YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java
FAILURE: Integrated in Hadoop-Hdfs-trunk #2204 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2204/)
YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2223 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2223/)
YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java
- hadoop-yarn-project/CHANGES.txt
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #266 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/266/)
YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #274 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/274/)
YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) (jeagles: rev 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
Pulled this into 2.6.1. The patch applies cleanly for the most part except for a couple of minor merge conflicts in test-cases which I fixed.
Ran compilation and TestClientRMService, TestRMContainerImpl, TestChildQueueOrder, TestLeafQueue, TestReservations, TestFifoScheduler before the push.
Use Case: A user launches an application on a secured cluster that runs for some time and then fails within the AM (perhaps due to OOM in the AM), leaving no history in the job history server. The user doesn't notice that the job has failed until after the application has dropped off of the RM's application store. At this point, if no information was stored in the Generic Application History Service, a user must rely on a priviledged system administrator to access the AM logs for them.
It is desirable to activate the Generic Application History service within the timeline server so that users can access their application's information even after the RM has forgotten about their application. This app information should be kept in the GAHS for 1 week, as is done, for example, for logs in the job history server.
One way that the Generic AHS stores metadata about an application is in an Entity levelDB. This includes information about each container for each application. Based on my analysis, the levelDB size grows by at least 2500 bytes per container (uncompressed). This is a conservative estimate as the size could be much bigger based on the amount of diagnostic information associated with failed containers.
On very large and busy clusters, the amount needed on the timeline server's local disk would be between 0.6 TB and 1.0 TB (uncompressed). Even if we assume 90% compression, that's still between 60 GB and 100 GB that will be needed on the local disk. In addition to this, between 80 GB and 143 GB of metadata (uncopressed) will need to be cleaned up every day from the levelDB, which will delay other processing in the timeline server.
The proposal of this JIRA is to add a configuration property that enables/disables whether or not the GAHS stores container information in the levelDB. Whith this change, I estimate that the local disk usage would be about 5700 bytes per job, or about 10 GB (uncompressed) per week. Additionally, the daily cleanup load would only be about 1.5 GB per day.