[HBASE-6060] Regions's in OPENING state from failed regionservers takes a long time to recover

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.95.0
• Component/s: master, regionserver
• Hadoop Flags: Reviewed
• Assignee: Jimmy Xiang
• Reporter: Enis Soztutar
• Created: 21/May/12 18:53
• Updated: 10/Jan/15 01:38
• Resolved: 01/Nov/12 20:53

Description

we have seen a pattern in tests, that the regions are stuck in OPENING state for a very long time when the region server who is opening the region fails. My understanding of the process:

• master calls rs to open the region. If rs is offline, a new plan is generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), HMaster.assign()
• RegionServer, starts opening a region, changes the state in znode. But that znode is not ephemeral. (see ZkAssign)
• Rs transitions zk node from OFFLINE to OPENING. See OpenRegionHandler.process()
• rs then opens the region, and changes znode from OPENING to OPENED
• when rs is killed between OPENING and OPENED states, then zk shows OPENING state, and the master just waits for rs to change the region state, but since rs is down, that wont happen.
• There is a AssignmentManager.TimeoutMonitor, which does exactly guard against these kind of conditions. It periodically checks (every 10 sec by default) the regions in transition to see whether they timedout (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, which explains what you and I are seeing.
• ServerShutdownHandler in Master does not reassign regions in OPENING state, although it handles other states.

Lowering that threshold from the configuration is one option, but still I think we can do better.

Will investigate more.

Issue Links

• duplicates: HBASE-7724 [0.94] Just OPENED regions are ignored by ServerShutdownHandler and go unassigned for ever
• is related to: HBASE-5396 Handle the regions in regionPlans while processing ServerShutdownHandler
• is related to: HBASE-6272 In-memory region state is inconsistent
• is related to: HBASE-7701 Opening regions on dead server are not reassigned quickly
• is related to: HBASE-5843 Improve HBase MTTR - Mean Time To Recover
• relates to: HBASE-5816 Balancer and ServerShutdownHandler concurrently reassign the same region
• relates to: HBASE-6198 Sometimes we synchronize on RegionState instance updating it, most of the time we don't. Fix
• relates to: HBASE-6199 Change PENDING_OPEN scope from pre-rpc open to OPENING to just post-rpc open to OPENING