[HBASE-11906] Meta data loss with distributed log replay

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.99.0, 2.0.0
• Fix Version/s: 0.99.1
• Hadoop Flags: Reviewed
• Assignee: Jeffrey Zhong
• Reporter: Jimmy Xiang
• Created: 06/Sep/14 00:30
• Updated: 06/Apr/18 17:55
• Resolved: 20/Sep/14 01:09

Description

In the attached log, you can see, before log replaying, the region is open on e1205:

A3. 2014-09-05 16:38:46,705 INFO  [B.defaultRpcServer.handler=5,queue=2,port=20020] master.RegionStateStore: Updating row IntegrationTestBigLinkedList,\x90Jy\x04\xA7\x90Jp,1409959495482.cbb0d736ebfabcf4a07e5a7b395fcdf7. with state=OPEN&openSeqNum=40118237&server=e1205.halxg.cloudera.com,20020,1409960280431

After the log replay, we got from meta the region is open on e1209

A4. 2014-09-05 16:41:12,257 INFO  [ActiveMasterManager] master.AssignmentManager: Loading from meta: {cbb0d736ebfabcf4a07e5a7b395fcdf7 state=OPEN, ts=1409960472257, server=e1209.halxg.cloudera.com,20020,1409959391651}

The replayed edits show the log does have the edit expected:

2014-09-05 16:41:11,862 INFO  [B.defaultRpcServer.handler=18,queue=0,port=20020] regionserver.RSRpcServices: Meta replay edit type=PUT,mutation={"totalColumns":4,"families":{"info":[{"timestamp":1409960326705,"tag":["3:\\x00\\x00\\x00\\x00\\x02bad"],"value":"e1205.halxg.cloudera.com:20020","qualifier":"server","vlen":30},{"timestamp":1409960326705,"tag":["3:\\x00\\x00\\x00\\x00\\x02bad"],"value":"\\x00\\x00\\x01HH.\\x81o","qualifier":"serverstartcode","vlen":8},{"timestamp":1409960326705,"tag":["3:\\x00\\x00\\x00\\x00\\x02bad"],"value":"\\x00\\x00\\x00\\x00\\x02d'\\xDD","qualifier":"seqnumDuringOpen","vlen":8},{"timestamp":1409960326706,"tag":["3:\\x00\\x00\\x00\\x00\\x02bad"],"value":"OPEN","qualifier":"state","vlen":4}]},"row":"IntegrationTestBigLinkedList,\\x90Jy\\x04\\xA7\\x90Jp,1409959495482.cbb0d736ebfabcf4a07e5a7b395fcdf7."}

Why we picked up a wrong value with an older time stamp?

2014-09-05 16:41:11,063 INFO  [B.defaultRpcServer.handler=9,queue=0,port=20020] regionserver.RSRpcServices: Meta replay edit type=PUT,mutation={"totalColumns":4,"families":{"info":[{"timestamp":1409959994634,"tag":["3:\\x00\\x00\\x00\\x00\\x00\\x00\\x09\\x99"],"value":"e1209.halxg.cloudera.com:20020","qualifier":"server","vlen":30},{"timestamp":1409959994634,"tag":["3:\\x00\\x00\\x00\\x00\\x00\\x00\\x09\\x99"],"value":"\\x00\\x00\\x01HH \\xF1\\xA3","qualifier":"serverstartcode","vlen":8},{"timestamp":1409959994634,"tag":["3:\\x00\\x00\\x00\\x00\\x00\\x00\\x09\\x99"],"value":"\\x00\\x00\\x00\\x00\\x00\\x01\\xB7\\xAB","qualifier":"seqnumDuringOpen","vlen":8},{"timestamp":1409959994634,"tag":["3:\\x00\\x00\\x00\\x00\\x00\\x00\\x09\\x99"],"value":"OPEN","qualifier":"state","vlen":4}]},"row":"IntegrationTestBigLinkedList,\\x90Jy\\x04\\xA7\\x90Jp,1409959495482.cbb0d736ebfabcf4a07e5a7b395fcdf7."}