[YARN-11277] trigger deletion of log-dir by size for NonAggregatingLogHandler

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: nodemanager
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Xianming Lei
• Reporter: Xianming Lei
• Created: 24/Aug/22 03:16
• Updated: 05/Jun/23 03:10
• Resolved: 05/Jun/23 03:08

Description

In our yarn cluster, the log files of some containers are too large, which causes the NodeManager to frequently switch to the unhealthy state. For logs that are too large, we can consider deleting them directly without delaying yarn.nodemanager.log.retain-seconds.

Cluster environment:

1. 8k nodes+
2. 50w+ apps  / day

Configuration:

1. yarn.nodemanager.log.retain-seconds=3days
2. yarn.log-aggregation-enable=false

 

 

Issue Links

• links to: GitHub Pull Request #4797
• links to: GitHub Pull Request #4797