[HBASE-12075] Preemptive Fast Fail - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Convert to Issue

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Sub-task
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 0.99.0, 0.98.6.1, 2.0.0
Fix Version/s: 0.99.2
Component/s: Client
Labels:
None

Hadoop Flags:

Reviewed
Release Note:

Hide
The Interceptor Framework can be used to modify the behavior of retrying client calls on the client side without altering the core of the client code. This framework gives reasonable flexibility with little intrusion.

An example use case of the interceptor is to have a custom sleep strategy within retries. We can consider a simple accounting system where we have a time spent quota on the client side for the servers. And we would want to error some of the calls if we have consumed the server's quota already.

Interceptor.intercept() {
// would keep track the start time.
server = getServer()
  // Throw if we already spent a lot of time on a server.
  timeSpentOnServer = getTimeSpentSinceLastReset(server)
  if ( timeSpentOnServer > THRESHOLD) throw new Exception("We spend too long on this server")
  setServerStartTime(server)
}

Interceptor.handleFailure() {
  server = getServer()
  updateTimeSpentOnServer(server)
}

Interceptor.updateFailureInfo() {
  server = getServer()
  updateTimeSpentOnServer(server)
}

Show
The Interceptor Framework can be used to modify the behavior of retrying client calls on the client side without altering the core of the client code. This framework gives reasonable flexibility with little intrusion. An example use case of the interceptor is to have a custom sleep strategy within retries. We can consider a simple accounting system where we have a time spent quota on the client side for the servers. And we would want to error some of the calls if we have consumed the server's quota already. Interceptor.intercept() { // would keep track the start time. server = getServer()   // Throw if we already spent a lot of time on a server.   timeSpentOnServer = getTimeSpentSinceLastReset(server)   if ( timeSpentOnServer > THRESHOLD) throw new Exception("We spend too long on this server")   setServerStartTime(server) } Interceptor.handleFailure() {   server = getServer()   updateTimeSpentOnServer(server) } Interceptor.updateFailureInfo() {   server = getServer()   updateTimeSpentOnServer(server) }

Description

In multi threaded clients, we use a feature developed on 0.89-fb branch called Preemptive Fast Fail. This allows the client threads which would potentially fail, fail fast. The idea behind this feature is that we allow, among the hundreds of client threads, one thread to try and establish connection with the regionserver and if that succeeds, we mark it as a live node again. Meanwhile, other threads which are trying to establish connection to the same server would ideally go into the timeouts which is effectively unfruitful. We can in those cases return appropriate exceptions to those clients instead of letting them retry.