[YARN-6288] Exceptions during aggregated log writes are mishandled

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.8.0, 2.7.3
• Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
• Component/s: log-aggregation
• Labels: supportability
• Target Version/s: 2.8.1
• Hadoop Flags: Reviewed
• Assignee: Akira Ajisaka
• Reporter: Akira Ajisaka
• Created: 06/Mar/17 05:05
• Updated: 23/Aug/17 15:15
• Resolved: 07/Apr/17 03:21

Description

In AppLogAggregatorImpl.java, if an exception occurs in writing container log to remote filesystem, the exception is not caught and ignored.

https://github.com/apache/hadoop/blob/f59e36b4ce71d3019ab91b136b6d7646316954e7/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java#L398

Issue Links

• breaks: YARN-7083 Log aggregation deletes/renames while file is open
• is related to: YARN-3760 FSDataOutputStream leak in AggregatedLogFormat.LogWriter.close()