[YARN-4697] NM aggregation thread pool is not bound by limits

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-alpha1
• Component/s: nodemanager
• Target Version/s: 2.8.0, 2.7.3, 2.6.4
• Hadoop Flags: Reviewed
• Assignee: Haibo Chen
• Reporter: Haibo Chen
• Created: 17/Feb/16 00:05
• Updated: 30/Aug/16 01:06
• Resolved: 24/Feb/16 23:05

Description

In the LogAggregationService.java we create a threadpool to upload logs from the nodemanager to HDFS if log aggregation is turned on. This is a cached threadpool which based on the javadoc is an ulimited pool of threads.
In the case that we have had a problem with log aggregation this could cause a problem on restart. The number of threads created at that point could be huge and will put a large load on the NameNode and in worse case could even bring it down due to file descriptor issues.

Issue Links

• is related to: YARN-4325 Nodemanager log handlers fail to send finished/failed events in some cases
• is related to: YARN-4984 LogAggregationService shouldn't swallow exception in handling createAppDir() which cause thread leak.