[HBASE-28385] Quota estimates are too optimistic for large scans

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0, 3.0.0-beta-2
• Labels: pull-request-available

Release Note

When hbase.quota.use.result.size.bytes is false, we will now estimate the amount of quota to grab for a scan based on the block bytes scanned of previous next() requests. This will increase throughput for large scans which might prefer to wait a little longer for a larger portion of the quota.

• Assignee: Ray Mattingly
• Reporter: Ray Mattingly
• Created: 20/Feb/24 19:52
• Updated: 14/Mar/24 14:41
• Resolved: 13/Mar/24 22:12

Description

Let's say you're running a table scan with a throttle of 100MB/sec per RegionServer. Ideally your scans are going to pull down large results, often containing hundreds or thousands of blocks.

You will estimate each scan as costing a single block of read capacity, and if your quota is already exhausted then the server will evaluate the backoff required for your estimated consumption (1 block) to be available. This will often be ~1ms, causing your retries to basically be immediate.

Obviously it will routinely take much longer than 1ms for 100MB of IO to become available in the given configuration, so your retries will be destined to fail. At worst this can cause a saturation of your server's RPC layer, and at best this causes erroneous exhaustion of the client's retries.

We should find a way to make these estimates a bit smarter for large scans.

Issue Links

• links to: GitHub Pull Request #5713