[YARN-221] NM should provide a way for AM to tell it not to aggregate logs.

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: log-aggregation, nodemanager
• Assignee: Ming Ma
• Reporter: Robert Joseph Evans
• Created: 16/Nov/12 18:45
• Updated: 07/Jul/22 11:42
• Resolved: 24/Aug/15 00:09

Description

The NodeManager should provide a way for an AM to tell it that either the logs should not be aggregated, that they should be aggregated with a high priority, or that they should be aggregated but with a lower priority. The AM should be able to do this in the ContainerLaunch context to provide a default value, but should also be able to update the value when the container is released.

This would allow for the NM to not aggregate logs in some cases, and avoid connection to the NN at all.

Issue Links

• blocks: YARN-386 [Umbrella] YARN API Changes
• breaks: YARN-4071 After YARN-221, ContainerLogsRetentionPolicy.java still remains in yarn-common causing compilation error
• breaks: YARN-4152 NM crash with NPE when LogAggregationService#stopContainer called for absent container
• duplicates: YARN-1377 Log aggregation via node manager should expose expose a way to cancel log aggregation at application or container level
• is related to: YARN-7087 NM failed to perform log aggregation due to absent container
• is related to: MAPREDUCE-6456 Support configurable log aggregation policy
• is related to: YARN-11208 SampleContainerLogAggregationPolicy is inconsistent with expectations
• relates to: YARN-85 Allow per job log aggregation configuration