[HBASE-23955] Have test runs use less resources

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: test
• Assignee: Unassigned
• Reporter: Michael Stack
• Created: 10/Mar/20 15:36
• Updated: 14/Mar/20 04:01
• Resolved: 14/Mar/20 03:40

Description

Our tests can create thousands of threads all up in the one JVM. Using less means less memory, less contention, and hopefully, likelier passes.

I've been studying the likes of TestNamespaceReplicationWithBulkLoadedData to see what it does as it runs (this test puts up 4 clusters with replication between). It peaks at 2k threads. After some configuration and using less HDFS, can get it down to ~800 threads and about 1/2 the memory-used.

(HDFS is a profligate offender. DataXceivers (Server and Client), jetty threads, Volume threads (async disk 'worker' then another for cleanup...), image savers, ipc clients – new thread per incoming connection w/o bound (or reuse), block responder threads, anonymous threads, and so on. Many are not configurable or boundable or are hard-coded; e.g. each volume gets 4 workers. Biggest impact was to be had by downing the count of data nodes. TODO: a follow-on that turns down DN counts in all tests)

I've been using Java Flight Recorder during this study. Here is how you get a flight recorder for the a single test run:

MAVEN_OPTS=" -XX:StartFlightRecording=disk=true,dumponexit=true,filename=recording.jfr,settings=profile,path-to-gc-roots=true,maxsize=1024m"  mvn  test -Dtest=TestNamespaceReplicationWithBulkLoadedData -Dsurefire.firstPartForkCount=0 -Dsurefire.secondPartForkCount=0 

i.e. start recording on mvn launch, bound the size of the recording, and have the test run in the mvn context (DON'T fork). Useful is connecting to the running test at the same time from JDK Mission Control. We do the latter because the thread reporting screen is overwhelmed by the count of running threads and if you connect live, you can at least get a 'live threads' graph w/ count as the test progresses. Useful.

When the test finishes, it dumps a .jfr file which can be opened in JDK MC. I've been compiling w/ JDK8 and then running w/ JDK11 so I can use JDK MC Version 7, the non-commercial latest. Works pretty well.

Let me put up a patch for tests that cuts down thread counts where we can.

 

 

Issue Links

• is duplicated by: HBASE-23956 Use less resources running tests