[YARN-2443] Handling logs of long-running services on YARN

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: nodemanager, resourcemanager
• Assignee: Xuan Gong
• Reporter: Xuan Gong
• Created: 22/Aug/14 17:24
• Updated: 24/Dec/14 02:20

Issue Links

• is related to: YARN-2985 YARN should support to delete the aggregated logs for Non-MapReduce applications
• is required by: YARN-896 Roll up for long-lived services in YARN

Sub-Tasks

• NMs to support rolling logs of stdout & stderr: Resolved
• Log handling for LRS: Closed
• Find a more scalable way to handle logs for long running service: Open
• API changes for handling logs of long-running services: Closed
• NMs need to find a way to get LogAggregationContext: Closed
• Log related CLI and Web UI changes for Aggregated Logs in LRS: Closed
• Modify the LogDeletionService to support Log aggregation for LRS: Closed
• Modify DistributedShell to enable LogAggregationContext: Resolved
• Spin off the LogRollingInterval from LogAggregationContext: Closed
• Improve CLI to fetch latest logs for running applications: Resolved
• Improve Web UI to fetch latest logs for Running applications: Resolved
• Add logUploadedTime into LogValue for better display: Closed
• containerLogNotFound log shows multiple time for the same container: Closed
• Misleading msg in LogCLI when app is not successfully submitted: Resolved
• Upload logs in the sub-folders under the local log dir when aggregating logs: Patch Available
• Should not upload partial logs for MR jobs or other "short-running' applications: Closed