[HBASE-13850] Check for dead server on CallTimeoutException

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.2.0, 2.0.0
• Component/s: Client, MTTR
• Assignee: Unassigned
• Reporter: Matteo Bertozzi
• Created: 05/Jun/15 18:44
• Updated: 24/Jun/22 19:17
• Resolved: 15/Nov/17 16:13

Description

WARN this may be a misconf, so let me know if there is a timeout param to set.

hbase-site.xml
zookeeper.session.timeout 10000
hbase.regionserver.storefile.refresh.period 10000
hbase.client.operation.timeout 5000
hbase.client.meta.operation.timeout 5000
hbase.client.scanner.timeout.period 10000
hbase.regionserver.lease.period 10000

I have a test that does a kill STOP on a RS and tries to query it.
From the conf the zk lease is 10sec, and the master is correctly doing the reassign after 10sec and meta is updated.

the client keep trying to query the RS for a specific row until it get a response. The table.get(row) in the loop throws a CallTimeoutException every 5sec (which is the configured settings). but instead of succeed after 2/3 retries (> 10sec where the master reassign) it keeps retrying up to 60sec (I don't know what that 60sec is, maybe a conf param that I'm not able to find)

one simple fix in the code is handling the CallTimeoutException in RegionServerCallable and clear the meta cache for that RS that is not responding. (but maybe there is already a conf to set to reduce that 60sec period)

Issue Links

• is related to: HBASE-15354 Use same criteria for clearing meta cache for all operations