[OOZIE-3600] LauncherAM yarn syslog empty due to non-zero yarn.app.container.log.filesize

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 5.1.0, 5.2.0
• Component/s: core
• Assignee: Unassigned
• Reporter: Bruno Dumon
• Created: 18/Jun/20 07:09
• Updated: 18/Jun/20 07:09

Description

I noticed that the yarn syslog of the job running a workflow with a java task was empty.

I tracked this down to yarn.app.container.log.filesize being set to a non-zero value in JavaActionExecutor#createCommand:

vargs.add("-D" + YarnConfiguration.YARN_APP_CONTAINER_LOG_SIZE + "=" + 1024 * 1024);

When this is non-zero, Hadoop's ContainerLogAppender will buffer a tail of log events and write nothing to the actual log file until close() is called on the ContainerLogAppender. However, close() is never called, hence the log stays empty.

For this to work, log4j's LogManager.shutdown() should be called. For example, in MapReduce's YarnChild, this is done where it calls TaskLog.syncLogsShutdown(). Oozie should do something similar.

But in fact, I don't see why by default only a tail of log events should be written, rather than all of them. So I would set yarn.app.container.log.filesize to 0 by default.

As a workaround, I'm passing -Dyarn.app.container.log.filesize=0 via java-opts in the workflow, which overwrites the value.