[YARN-2942] Aggregated Log Files should be combined

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.6.0
• Assignee: Robert Kanter
• Reporter: Robert Kanter
• Created: 09/Dec/14 23:27
• Updated: 17/Oct/16 21:45
• Resolved: 17/Oct/16 21:45

Description

Turning on log aggregation allows users to easily store container logs in HDFS and subsequently view them in the YARN web UIs from a central place. Currently, there is a separate log file for each Node Manager. This can be a problem for HDFS if you have a cluster with many nodes as you’ll slowly start accumulating many (possibly small) files per YARN application. The current “solution” for this problem is to configure YARN (actually the JHS) to automatically delete these files after some amount of time.

We should improve this by compacting the per-node aggregated log files into one log file per application.

Issue Links

• is related to: MAPREDUCE-6375 Modify the JHS to be able to read the ConcatenatableAggregatedLogFormat
• is related to: MAPREDUCE-6415 Create a tool to combine aggregated logs into HAR files
• relates to: YARN-2548 Find a more scalable way to handle logs for long running service
• requires: YARN-1376 NM need to notify the log aggregation status to RM through Node heartbeat
• requires: HDFS-3689 Add support for variable length block

Sub-Tasks

• Implement ConcatenatableAggregatedLogFormat Reader and Writer: Resolved
• Modify the NM to write logs using the ConcatenatableAggregatedLogFormat: Resolved
• Create a Service in the RM to concatenate aggregated logs: Resolved
• Modify the yarn CLI to be able to read the ConcatenatableAggregatedLogFormat: Resolved
• Add an rmadmin command to compact concatenated aggregated logs: Resolved