[HBASE-23956] Use less resources running tests

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.3.0
• Component/s: test
• Hadoop Flags: Reviewed
• Assignee: Michael Stack
• Reporter: Michael Stack
• Created: 10/Mar/20 15:40
• Updated: 09/Jul/22 07:45
• Resolved: 29/Apr/20 22:44

Description

Our tests can create thousands of threads all up in the one JVM. Using less means less memory, less contention, likelier passes, and later, more possible parallelism.

I've been studying the likes of TestNamespaceReplicationWithBulkLoadedData to see what it does as it runs (this test puts up 4 clusters with replication between). It peaks at 2k threads. After some configuration and using less HDFS, its possible to get it down to ~800 threads and about 1/2 the memory-used. HDFS is a main offender. DataXceivers (Server and Client), jetty threads, Volume threads (async disk 'worker' then another for cleanup...), image savers, ipc clients – new thread per incoming connection w/o bound (or reuse), block responder threads, anonymous threads, and so on. Many are not configurable or boundable or are hard-coded; e.g. each volume gets 4 workers regardless. Biggest impact was just downing the count of data nodes. TODO: a follow-on that turns down DN counts in all tests.

I've been using Java Flight Recorder during this study. Here is how you get a flight recorder for the a single test run: {code:java} MAVEN_OPTS=" -XX:StartFlightRecording=disk=true,dumponexit=true,filename=recording.jfr,settings=profile,path-to-gc-roots=true,maxsize=1024m" mvn test -Dtest=TestNamespaceReplicationWithBulkLoadedData -Dsurefire.firstPartForkCount=0 -Dsurefire.secondPartForkCount=0 {code} i.e. start recording on mvn launch, bound the size of the recording, and have the test run in the mvn context (DON'T fork). Useful is connecting to the running test at the same time from JDK Mission Control. We do the latter because the thread reporting screen is overwhelmed by the count of running threads and if you connect live, you can at least get a 'live threads' graph w/ count as the test progresses. Useful. When the test finishes, it dumps a .jfr file which can be opened in JDK MC.

I've been compiling w/ JDK8 and then running w/ JDK11 so I can use JDK MC Version 7, the non-commercial latest. Works pretty well. Let me put up a patch for tests that cuts down thread counts where we can.

Let me put up a patch that does first pass on curtailing resource usage.

Issue Links

• Blocked: HBASE-26029 It is not reliable to use nodeDeleted event to track region server's death
• breaks: HBASE-23987 NettyRpcClientConfigHelper will not share event loop by default which is incorrect
• breaks: HBASE-26039 TestReplicationKillRS is useless after HBASE-23956
• duplicates: HBASE-23955 Have test runs use less resources
• is duplicated by: HBASE-23882 Scale *MiniCluster config for the environment it runs in.
• is related to: HBASE-24847 Backport HBASE-23956: Use less resources running tests to branch-2.2
• relates to: HBASE-24646 Set the log level for ScheduledChore to INFO in HBTU
• links to: GitHub Pull Request #1266