[YARN-10781] The Thread of the NM aggregate log is exhausted and no other Application can aggregate the log

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.9.2, 3.3.0
• Component/s: yarn
• Assignee: Unassigned
• Reporter: Xiping Zhang
• Created: 21/May/21 05:37
• Updated: 26/May/21 03:21

Description

We observed more than 100 applications running on one NM.Most of these applications are SparkStreaming applications, but these applications do not have running Containers.When the offline application running on it finishes, the log cannot be reported to HDFS. When we killed a large number of SparkStreaming applications, we found that a large number of log files were being created on the NN side, causing the read and write performance on the NN side to degrade significantly.Causes the business application to time out。