[YARN-9927] RM multi-thread event processing mechanism

• Type: Sub-task
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0, 2.9.2
• Component/s: yarn
• Target Version/s: 2.9.2
• Assignee: Qi Zhu
• Reporter: hcarrot
• Created: 22/Oct/19 06:41
• Updated: 24/May/21 11:51

Description

Recently, we have observed serious event blocking in RM event dispatcher queue. After analysis of RM event monitoring data and RM event processing logic, we found that

1) environment: a cluster with thousands of nodes

2) RMNodeStatusEvent dominates 90% time consumption of RM event scheduler

3) Meanwhile, RM event processing is in a single-thread mode, and It results in the low headroom of RM event scheduler, thus performance of RM.

So we proposed a RM multi-thread event processing mechanism to improve RM performance.

Issue Links

• Blocked: YARN-10784 Optimize RMNodeStatusEvent process performance
• duplicates: YARN-9926 RM multi-thread event processing mechanism
• is related to: YARN-9615 Add dispatcher metrics to RM
• is related to: YARN-9871 Miscellaneous scalability improvement