[MAPREDUCE-1648] Use rolling to limit tasklogs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: tasktracker
• Labels: logging
• Assignee: Unassigned
• Reporter: Guilin Sun
• Created: 30/Mar/10 12:55
• Updated: 30/Jul/14 17:22
• Resolved: 30/Jul/14 17:22

Description

There are at least two types of task-logs: syslog and stdlog

Task-Jvm outputs syslog by log4j with TaskLogAppender, TaskLogAppender looks just like "tail -c", it stores last N byte/line logs in memory(via queue), and do real output only if all logs is commit and Appender is going to close.

The common problem of TaskLogAppender and 'tail -c' is keep everything in memory and user can't see any log output while task is in progress.

So I'm going to try RollingFileAppender instead of TaskLogAppender, use MaxFileSize&MaxBackupIndex to limit log file size.

RollingFileAppender is also suitable for stdout/stderr, just redirect stdout/stderr to log4j via LoggingOutputStream, no client code have to be changed, and RollingFileAppender seems better than 'tail -c' too.

Issue Links

• is depended upon by: MAPREDUCE-1799 TaskTracker webui fails to show logs for tasks whose child JVM itself crashes before process launch
• is related to: MAPREDUCE-1716 Truncate logs of finished tasks to prevent node thrash due to excessive logging