[HBASE-14205] RegionCoprocessorHost System.nanoTime() performance bottleneck

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.2.0, 1.3.0, 2.0.0
• Component/s: Coprocessors, Performance, regionserver
• Labels: needs_releasenote
• Hadoop Flags: Incompatible change, Reviewed
• Assignee: Andrew Kyle Purtell
• Reporter: Jan Van Besien
• Created: 11/Aug/15 08:44
• Updated: 01/Jul/22 20:49
• Resolved: 18/Dec/15 00:29

Description

The tracking of execution time of coprocessor methods introduced in HBASE-11516 introduces 2 calls to System.nanoTime() per coprocessor method per coprocessor. This is resulting in a serious performance bottleneck in certain scenarios.

For example consider the scenario where many rows are being ingested (PUT) in a table which has multiple coprocessors (we have up to 20 coprocessors). This results in 8 extra calls to System.nanoTime() per coprocessor (prePut, postPut, postStartRegionOperation and postCloseRegionOperation) which has in total (i.e. times 20) been seen to result in a 50% increase of execution time.

I think it is generally considered bad practice to measure execution times on such a small scale (per single operation). Also note that measurements are taken even for coprocessors that do not even have an actual implementation for certain operations, making the problem worse.

Issue Links

• supercedes: HBASE-13420 RegionEnvironment.offerExecutionLatency Blocks Threads under Heavy Load

Sub-Tasks

• Remove documentation for coprocessor execution times after HBASE-14205: Resolved