[HBASE-7701] Opening regions on dead server are not reassigned quickly

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.95.2
• Fix Version/s: 0.95.0
• Hadoop Flags: Reviewed
• Assignee: Jimmy Xiang
• Reporter: Sergey Shelukhin
• Created: 28/Jan/13 23:45
• Updated: 23/Sep/13 18:31
• Resolved: 01/Feb/13 18:17

Description

Closed regions are not removed from assignments. I am not sure if it's a general state problem, or just a small bug; for now, one manifestation is that moved region is ignored by SSH of the target server if target server dies before updating ZK.

2013-01-22 17:59:00,524 DEBUG [IPC Server handler 3 on 50658] master.AssignmentManager(1475): Sent CLOSE to 10.11.2.92,51231,1358906285048 for region IntegrationTestRebalanceAndKillServersTargeted,66666660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb.
2013-01-22 17:59:00,997 DEBUG [RS_CLOSE_REGION-10.11.2.92,51231,1358906285048-1] handler.CloseRegionHandler(167): set region closed state in zk successfully for region IntegrationTestRebalanceAndKillServersTargeted,66666660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb. sn name: 10.11.2.92,51231,1358906285048
2013-01-22 17:59:01,088 INFO  [MASTER_CLOSE_REGION-10.11.2.92,50658,1358906192673-0] master.RegionStates(242): Region {NAME => 'IntegrationTestRebalanceAndKillServersTargeted,66666660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb.', STARTKEY => '66666660', ENDKEY => '7333332c', ENCODED => 0200b366bc37c5afd1185f7d487c7dfb,} transitioned from {IntegrationTestRebalanceAndKillServersTargeted,66666660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb. state=CLOSED, ts=1358906341087, server=null} to {IntegrationTestRebalanceAndKillServersTargeted,66666660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb. state=OFFLINE, ts=1358906341088, server=null}
2013-01-22 17:59:01,128 INFO  [MASTER_CLOSE_REGION-10.11.2.92,50658,1358906192673-0] master.AssignmentManager(1596): Assigning region IntegrationTestRebalanceAndKillServersTargeted,66666660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb. to 10.11.2.92,50661,1358906192942

... (50661 didn't update ZK to OPEN, only OPENING)

2013-01-22 17:59:06,605 INFO  [MASTER_SERVER_OPERATIONS-10.11.2.92,50658,1358906192673-2] handler.ServerShutdownHandler(202): Reassigning 7 region(s) that 10.11.2.92,50661,1358906192942 was carrying (skipping 0 regions(s) that are already in transition)
2013-01-22 17:59:06,605 DEBUG [MASTER_SERVER_OPERATIONS-10.11.2.92,50658,1358906192673-2] handler.ServerShutdownHandler(219): Skip assigning region IntegrationTestRebalanceAndKillServersTargeted,66666660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb. because it has been opened in 10.11.2.92,51231,1358906285048

Note the server in the last line - the one that has long closed the region.

Issue Links

• relates to: HBASE-7520 org.apache.hadoop.hbase.IntegrationTestRebalanceAndKillServersTargeted fails when I cd hbase-it and mvn verify
• relates to: HBASE-6060 Regions's in OPENING state from failed regionservers takes a long time to recover