[YARN-2468] Log handling for LRS

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0
• Component/s: log-aggregation, nodemanager, resourcemanager
• Target Version/s: 2.6.0
• Hadoop Flags: Reviewed
• Assignee: Xuan Gong
• Reporter: Xuan Gong
• Created: 29/Aug/14 01:29
• Updated: 25/Jul/16 11:02
• Resolved: 03/Oct/14 19:21

Description

Currently, when application is finished, NM will start to do the log aggregation. But for Long running service applications, this is not ideal. The problems we have are:
1) LRS applications are expected to run for a long time (weeks, months).
2) Currently, all the container logs (from one NM) will be written into a single file. The files could become larger and larger.

Issue Links

• blocks: SLIDER-116 Log handling for long-lived applications
• duplicates: YARN-1104 NMs to support rolling logs of stdout & stderr
• is related to: YARN-1111 NM containerlogs servlet can't handle logs of more than a GB
• is related to: YARN-3089 LinuxContainerExecutor does not handle file arguments to deleteAsUser
• is required by: YARN-2582 Log related CLI and Web UI changes for Aggregated Logs in LRS
• is required by: YARN-2583 Modify the LogDeletionService to support Log aggregation for LRS
• relates to: TWILL-23 Log to a separate file
• requires: YARN-2569 API changes for handling logs of long-running services
• requires: YARN-2581 NMs need to find a way to get LogAggregationContext