[HADOOP-16850] Support getting thread info from thread group for JvmMetrics to improve the performance

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.6, 2.9.3, 3.1.4, 3.2.2, 2.10.1, 3.3.1
• Fix Version/s: 3.3.0
• Component/s: metrics, performance
• Target Version/s: 3.3.1
• Hadoop Flags: Reviewed
• Release Note: Add a new configuration "hadoop.metrics.jvm.use-thread-mxbean". If set to true, ThreadMXBean is used for getting thread info in JvmMetrics, otherwise, ThreadGroup is used. The default value is false (use ThreadGroup for better performance).
• Assignee: Tao Yang
• Reporter: Tao Yang
• Created: 11/Feb/20 04:54
• Updated: 14/Feb/20 08:22
• Resolved: 14/Feb/20 07:19

Description

Recently we found jmx request taken almost 5s+ to be done when there were 1w+ threads in a stressed datanode process, meanwhile other http requests were blocked and some disk operations were affected (we can see many "Slow manageWriterOsCache" messages in DN log, and these messages were hard to be seen again after we stopped sending jxm requests)

The excessive time is spent in getting thread info via ThreadMXBean inside which ThreadImpl#getThreadInfo native method is called, the time complexity of ThreadImpl#getThreadInfo is O(n*n) according to JDK-8185005 and it holds global thread lock and prevents creation or termination of threads.

To improve this, I propose to support getting thread info from thread group which will improve a lot by default, also support using original approach when "-Dhadoop.metrics.jvm.use-thread-mxbean=true" is configured in the startup command.

An example of performance tests between these two approaches is as follows:

#Threads=100, ThreadMXBean=382372 ns, ThreadGroup=72046 ns, ratio: 5
#Threads=200, ThreadMXBean=776619 ns, ThreadGroup=83875 ns, ratio: 9
#Threads=500, ThreadMXBean=3392954 ns, ThreadGroup=216269 ns, ratio: 15
#Threads=1000, ThreadMXBean=9475768 ns, ThreadGroup=220447 ns, ratio: 42
#Threads=2000, ThreadMXBean=53833729 ns, ThreadGroup=579608 ns, ratio: 92
#Threads=3000, ThreadMXBean=196829971 ns, ThreadGroup=1157670 ns, ratio: 170