[YARN-9861] The ResourceManager log reports an error "Too many open files", the analysis is related to the service

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Component/s: yarn-native-services

Environment

yarn version:3.3.0-SNAPSHOT
hdfs version:2.7.1

• Assignee: Unassigned
• Reporter: huiyangjian
• Created: 27/Sep/19 10:18
• Updated: 08/Nov/19 10:44
• Resolved: 08/Nov/19 10:44

Description

The ResourceManager log outputs "Too many open files" and cannot commit a new task.
1. First is the error in picture1,
2. Then check the file handle open by RM (lsof -p PID), see picture 2,
3. Also read nameNode audit log (picture 3),
4. Confirm about service according to the path of service configuration (picture 4),
5. Handle number growth trend (picture 5).