[HBASE-20864] RS was killed due to master thought the region should be on a already dead server

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Resolved
• Affects Version/s: 2.0.0
• Assignee: Allan Yang
• Reporter: Allan Yang
• Created: 10/Jul/18 08:46
• Updated: 23/Jun/22 19:28
• Resolved: 20/Jul/18 02:00

Description

When I was running ITBLL with our internal 2.0.0 version(with 2.0.1 backported and with other two issues: HBASE-20706, HBASE-20752). I found two of my RS killed by master since master has a different region state with those RS. It is very strange that master thought these region should be on a already dead server. There might be a serious bug, but I haven't found it yet. Here is the process:

1. e010125048153.bja,60020,1531137365840 is crashed, and clearly 4423e4182457c5b573729be4682cc3a3 was assigned to e010125049164.bja,60020,1531136465378 during ServerCrashProcedure

2018-07-09 20:03:32,443 INFO  [PEWorker-10] procedure.ServerCrashProcedure: Start pid=2303, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=e010125048153.bja,60020,1531137365840, splitWal=true, meta=false
2018-07-09 20:03:39,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=294,queue=24,port=60000] assignment.RegionTransitionProcedure: Received report OPENED seqId=16021, pid=2305, ppid=2303, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=IntegrationTestBigLinkedList, region=4423e4182457c5b573729be4682cc3a3; rit=OPENING, location=e010125049164.bja,60020,1531136465378
2018-07-09 20:03:39,220 INFO  [PEWorker-13] assignment.RegionStateStore: pid=2305 updating hbase:meta row=4423e4182457c5b573729be4682cc3a3, regionState=OPEN, openSeqNum=16021, regionLocation=e010125049164.bja,60020,1531136465378
2018-07-09 20:03:43,190 INFO  [PEWorker-12] procedure2.ProcedureExecutor: Finished pid=2303, state=SUCCESS; ServerCrashProcedure server=e010125048153.bja,60020,1531137365840, splitWal=true, meta=false in 10.7490sec

2. A modify table happened later, the 4423e4182457c5b573729be4682cc3a3 was reopend on e010125049164.bja,60020,1531136465378

2018-07-09 20:04:39,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=295,queue=25,port=60000] assignment.RegionTransitionProcedure: Received report OPENED seqId=16024, pid=2351, ppid=2314, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=IntegrationTestBigLinkedList, region=4423e4182457c5b573729be4682cc3a3, target=e010125049164.bja,60020,1531136465378; rit=OPENING, location=e010125049164.bja,60020,1531136465378
2018-07-09 20:04:40,554 INFO  [PEWorker-6] assignment.RegionStateStore: pid=2351 updating hbase:meta row=4423e4182457c5b573729be4682cc3a3, regionState=OPEN, openSeqNum=16024, regionLocation=e010125049164.bja,60020,1531136465378

3. Active master was killed, the backup master took over, but when loading meta entry, it clearly showed 4423e4182457c5b573729be4682cc3a3 is on the privous dead server e010125048153.bja,60020,1531137365840. That is very very strange!!!

2018-07-09 20:06:17,985 INFO  [master/e010125048016:60000] assignment.RegionStateStore: Load hbase:meta entry region=4423e4182457c5b573729be4682cc3a3, regionState=OPEN, lastHost=e010125049164.bja,60020,1531136465378, regionLocation=e010125048153.bja,60020,1531137365840, openSeqNum=16024

4. the rs was killed

2018-07-09 20:06:20,265 WARN  [RpcServer.default.FPBQ.Fifo.handler=297,queue=27,port=60000] assignment.AssignmentManager: Killing e010125049164.bja,60020,1531136465378: rit=OPEN, location=e010125048153.bja,60020,1531137365840, table=IntegrationTestBigLinkedList, region=4423e4182457c5b573729be4682cc3a3reported OPEN on server=e010125049164.bja,60020,1531136465378 but state has otherwise.

Issue Links

• relates to: HBASE-20903 backport HBASE-20792 "info:servername and info:sn inconsistent for OPEN region" to branch-2.0
• relates to: HBASE-20792 info:servername and info:sn inconsistent for OPEN region