[YARN-11588] Fix uncleaned threads in YARN Federation interceptor threadpool

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: federation, router
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Jeffrey Chang
• Reporter: Jeffrey Chang
• Created: 09/Oct/23 08:09
• Updated: 25/Jan/24 14:49
• Resolved: 12/Oct/23 11:15

Description

In YARN Federation a new interceptor is generated for each new user that calls YARN Router. This also generates a threadpool which is used to call YARN RM. However, when the call is finished the threadpool threads never exits, causing the interceptor object for the user to never be cleaned up. This causes an issue when the number of users is large in the YARN cluster, as the number of idle, uncleaned thread can reach # of users calling router times number of active core threads in thread pool. This causes performance issues in router due to too many idle threads. 

Issue Links

• links to: GitHub Pull Request #6159
• links to: GitHub Pull Request #6222