[YARN-6875] New aggregated log file format for YARN log aggregation.

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Xuan Gong
• Reporter: Xuan Gong
• Created: 26/Jul/17 21:17
• Updated: 12/Jun/19 06:55
• Resolved: 12/Jun/19 06:55

Description

T-file is the underlying log format for the aggregated logs in YARN. We have seen several performance issues, especially for very large log files.
We will introduce a new log format which have better performance for large log files.

Sub-Tasks

• Create an abstract log writer for extendability: Resolved
• Create an abstract log reader for extendability: Resolved
• Add a new log aggregation file format controller: Resolved
• Mark the log-aggregation-controller APIs as public: Resolved
• Log Aggregation controller should not swallow the exceptions when it calls closeWriter and closeReader.: Resolved
• TestLogsCLI fails in trunk and branch-2 and javadoc error: Resolved
• Log aggregation status is always Failed with the newly added log aggregation IndexedFileFormat: Resolved
• Add size-based rolling policy to LogAggregationIndexedFileController: Resolved
• re-factory IndexedFileAggregatedLogsBlock and TFileAggregatedLogsBlock to remove duplicate codes: Resolved
• Make HAR tool support IndexedLogAggregtionController: Resolved
• LogAggregationIndexedFileController should support read from HAR file: Resolved
• IFile format is not working against s3a remote folder: Patch Available
• Auto-configuring rollover-size of IFile format for non-appendable filesystems: Patch Available