[HBASE-3984] CT.verifyRegionLocation isn't doing a very good check, can delay cluster recovery

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.90.3
• Fix Version/s: 0.90.4
• Hadoop Flags: Reviewed

Release Note

In trunk:
All HRegionInferface methods will now throw a RegionServerStoppedException if it's in that state, whereas we used to only check it for a few methods.
SingleServerBulkAssigner will not kill the Master anymore when getting IOEs, instead it will just log an error and the TimeoutMonitor will take care of picking up the pieces.

In 0.90:
Only a couple of checkOpen calls were added in order to change as less code as possible while still fixing the issue.

• Assignee: Jean-Daniel Cryans
• Reporter: Jean-Daniel Cryans
• Created: 13/Jun/11 18:36
• Updated: 20/Nov/15 11:53
• Resolved: 29/Jun/11 22:29

Description

After some extensive debugging in the thread A sudden msg of "java.io.IOException: Server not running, aborting", we figured that the region servers weren't able to talk to the new .META. location because the old one was still alive but on it's way down after a OOME.

It translates into exceptions like "Server not running" coming from trying to edit .META. and digging in the code I see that CT.waitForMetaServerConnectionDefault -> waitForMeta -> getMetaServerConnection(true) calls verifyRegionLocation since we force the refresh. In this method we check if the RS is good by calling getRegionInfo which does not check if the region server is trying to close.

What this means is that a cluster can't recover a .META.-serving RS failure until it has fully shutdown since every time a RS tries to open a region (like right after the log splitting) or split it fails editing .META.