[YARN-7083] Log aggregation deletes/renames while file is open

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.8.2
• Fix Version/s: 2.8.2
• Component/s: nodemanager
• Target Version/s: 2.8.2
• Hadoop Flags: Reviewed
• Assignee: Jason Darrell Lowe
• Reporter: Daryn Sharp
• Created: 23/Aug/17 15:15
• Updated: 29/Aug/17 20:24
• Resolved: 29/Aug/17 20:24

Description

YARN-6288 changes the log aggregation writer to be an autoclosable. Unfortunately the try-with-resources block for the writer will either rename or delete the log while open.

Assuming the NM's behavior is correct, deleting open files only results in ominous WARNs in the nodemanager log and increases the rate of logging in the NN when the implicit try-with-resource close fails. These red herrings complicate debugging efforts.

Issue Links

• is broken by: YARN-6288 Exceptions during aggregated log writes are mishandled
• is cloned by: YARN-7124 LogAggregationTFileController deletes/renames while file is open