[HBASE-28359] Improve quota RateLimiter synchronization

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0, 3.0.0-beta-2
• Labels: pull-request-available
• Assignee: Ray Mattingly
• Reporter: Bryan Beaudreault
• Created: 13/Feb/24 13:22
• Updated: 07/Mar/24 21:01
• Resolved: 07/Mar/24 02:35

Description

We've been experiencing RpcThrottlingException with 0ms waitInterval. This seems odd and wasteful, since the client side will immediately retry without backoff. I think the problem is related to the synchronization of RateLimiter.

The TimeBasedLimiter checkQuota method does the following:

if (!reqSizeLimiter.canExecute(estimateWriteSize + estimateReadSize)) {
  RpcThrottlingException.throwRequestSizeExceeded(
    reqSizeLimiter.waitInterval(estimateWriteSize + estimateReadSize));
} 

Both canExecute and waitInterval are synchronized, but we're calling them independently. So it's possible under high concurrency for canExecute to return false, but then waitInterval returns 0 (would have been true)

I think we should simplify the API to have a single synchronized call:

long waitInterval = reqSizeLimiter.tryAcquire(estimateWriteSize + estimateReadSize);
if (waitInterval > 0) {
  RpcThrottlingException.throwRequestSizeExceeded(waitInterval);
}

Issue Links

• links to: GitHub Pull Request #5683