[HBASE-5162] Basic client pushback mechanism

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.92.0
• Fix Version/s: 2.0.0
• Assignee: Jesse Yates
• Reporter: Jean-Daniel Cryans
• Created: 10/Jan/12 00:35
• Updated: 12/Jun/22 20:04
• Resolved: 17/Dec/14 20:48

Description

The current blocking we do when we are close to some limits (memstores over the multiplier factor, too many store files, global memstore memory) is bad, too coarse and confusing. After hitting HBASE-5161, it really becomes obvious that we need something better.

I did a little brainstorm with Stack, we came up quickly with two solutions:

• Send some exception to the client, like OverloadedException, that's thrown when some situation happens like getting past the low memory barrier. It would be thrown when the client gets a handler and does some check while putting or deleting. The client would treat this a retryable exception but ideally wouldn't check .META. for a new location. It could be fancy and have multiple levels of pushback, like send the exception to 25% of the clients, and then go up if the situation persists. Should be "easy" to implement but we'll be using a lot more IO to send the payload over and over again (but at least it wouldn't sit in the RS's memory).
• Send a message alongside a successful put or delete to tell the client to slow down a little, this way we don't have to do back and forth with the payload between the client and the server. It's a cleaner (I think) but more involved solution.

In every case the RS should do very obvious things to notify the operators of this situation, through logs, web UI, metrics, etc.

Other ideas?

Issue Links

• is duplicated by: HBASE-5333 Introduce Memstore "backpressure" for writes
• is related to: HBASE-6423 Writes should not block reads on blocking updates to memstores
• is related to: HBASE-12731 Heap occupancy based client pushback
• is related to: HBASE-12911 Client-side metrics
• is related to: HBASE-2981 Improve consistency of performance during heavy load
• relates to: HBASE-22267 Implement client push back for async client
• relates to: HBASE-12703 Cleanup TestClientPushback for repeatability
• relates to: HBASE-12217 System load average based client pushback
• relates to: HBASE-12840 Improve unit test coverage of the client pushback mechanism
• relates to: HBASE-12841 ClientBackoffPolicies should support immediate rejection of submitted ops
• relates to: HBASE-12986 Compaction pressure based client pushback
• relates to: HBASE-14693 Add client-side metrics for received pushback signals
• relates to: HBASE-14756 Break out ClientBackoffPolicy factors into configurable and composable components
• relates to: HBASE-12906 Document the new load pushback mechanism and available policies
• relates to: HBASE-12702 Backport client pushback