[YARN-6875] New aggregated log file format for YARN log aggregation.

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Xuan Gong
• Reporter: Xuan Gong
• Created: 26/Jul/17 21:17
• Updated: 08/Aug/17 19:48

Description

T-file is the underlying log format for the aggregated logs in YARN. We have seen several performance issues, especially for very large log files.
We will introduce a new log format which have better performance for large log files.

Sub-Tasks

• Create an abstract log writer for extendability: Resolved
• Create an abstract log reader for extendability: Resolved
• Add a new log aggregation file format controller: Resolved
• Mark the log-aggregation-controller APIs as public: Resolved
• Log Aggregation controller should not swallow the exceptions when it calls closeWriter and closeReader.: Resolved
• TestLogsCLI fails in trunk and branch-2 and javadoc error: Resolved
• Log aggregation status is always Failed with the newly added log aggregation IndexedFileFormat: Resolved
• Add size-based rolling policy to LogAggregationIndexedFileController: Resolved
• re-factory IndexedFileAggregatedLogsBlock and TFileAggregatedLogsBlock to remove duplicate codes: Resolved
• Make HAR tool support IndexedLogAggregtionController: Resolved
• LogAggregationIndexedFileController should support read from HAR file: Resolved