[YARN-1130] Improve the log flushing for tasks when mapred.userlog.limit.kb is set

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.0.5-alpha
• Component/s: nodemanager
• Assignee: Paul Han
• Reporter: Paul Han
• Created: 30/Aug/13 22:16
• Updated: 28/Sep/15 21:09
• Resolved: 01/Feb/14 03:20

Description

When userlog limit is set with something like this:

<property>
<name>mapred.userlog.limit.kb</name>
<value>2048</value>
<description>The maximum size of user-logs of each task in KB. 0 disables the cap.
</description>
</property>

the log entry will be truncated randomly for the jobs.

The log size is left between 1.2MB to 1.6MB.

Since the log is already limited, avoid the log truncation is crucial for user.

The other issue with the current impl(org.apache.hadoop.yarn.ContainerLogAppender) is that log entries will not flush to file until the container shutdown and logmanager close all appenders. If user likes to see the log during task execution, it doesn't support it.

Will propose a patch to add a flush mechanism and also flush the log when task is done.

Issue Links

• duplicates: MAPREDUCE-5693 Restore MRv1 behavior for log flush
• duplicates: MAPREDUCE-5672 Provide optional RollingFileAppender for container log4j (syslog)
• is related to: MAPREDUCE-5672 Provide optional RollingFileAppender for container log4j (syslog)