Details
-
Type:
Sub-task
-
Status: Open
-
Priority:
Critical
-
Resolution: Unresolved
-
Affects Version/s: None
-
Fix Version/s: None
-
Component/s: nodemanager
-
Labels:None
Description
Currently, if log aggregation is enabled for a cluster - logs for all jobs will be aggregated - leading to a whole bunch of files on hdfs which users may not want.
Users should be able to control this along with the aggregation policy - failed only, all, etc.
Issue Links
- blocks
-
YARN-386
[Umbrella] YARN API Changes
-
- Open
-
- is related to
-
YARN-221
NM should provide a way for AM to tell it not to aggregate logs.
-
- Patch Available
-
-
MAPREDUCE-3143
Complete aggregation of user-logs spit out by containers onto DFS
-
- Closed
-
Activity
In the initial patch, am planning to only support per job enable/disable aggregation. Along with this - an option to keep the logs on the NM for some time if the job disables aggregation.
Something like FAILED_ONLY will not work very well rightnow - since container exit status is not necessarily an indication of whether a task completes successfully or not.
Once YARN-221 goes in we should be able to expose a lot more functionality, like aggregate only a sampling of successful jobs, aggregate only failed jobs, etc.
I have submitted a patch on YARN-221 and I am also starting to work on the MapReduce level changes to provide configuration for log aggregation on a per job basis. I will hopefully post a patch soon.
Patch looks good to me. Can anyone else also take a look at the patch.
Regarding Seth's comment of "container exit status is not necessarily an indication of whether a task completes successfully or not", https://issues.apache.org/jira/browse/MAPREDUCE-5465 should fix the issue.
+1, the infrastructure for this is all there, including tests IIRC, we just need to expose it to the users.