[ACCUMULO-1113] consistency check failure on !METADATA table

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Component/s: master, tserver
• Labels: 15_qa_bug

Environment

10 node test cluster running random walk test

• Assignee: Eric C. Newton
• Reporter: Eric C. Newton
• Created: 26/Feb/13 20:13
• Updated: 25/Apr/14 18:04
• Resolved: 12/Apr/13 19:15

Description

On nodeA:

2013-02-26 19:44:49,184 [tabletserver.MinorCompactor] DEBUG: Begin minor compaction /accumulo/tables/!0/root_tablet/F000221z.rf_tmp !0;!0<<
2013-02-26 19:44:49,207 [tabletserver.Tablet] TABLET_HIST: !0;!0<< MinC [memory] -> /root_tablet/F000221z.rf
2013-02-26 19:44:49,267 [tabletserver.TabletServer] DEBUG: Unassigning !0;!0<<@(null,10.0.0.8:9997[33cf2816b97c139],null)
2013-02-26 19:44:49,267 [tabletserver.TabletServer] DEBUG: Failed to unload tablet !0;!0<<... it was alread closing or closed : Tablet !0;!0<< already closed closing closeComplete
2013-02-26 19:44:49,267 [tabletserver.TabletServer] DEBUG: Failed to unload tablet !0;!0<<... it was alread closing or closed : Tablet !0;!0<< already closed closing closeComplete
2013-02-26 19:44:49,267 [state.ZooStore] DEBUG: Removing /root_tablet/location

On nodeB, the consistency check fails just a few seconds later:

2013-02-26 19:44:49,283 [tabletserver.TabletServer] INFO : Loading tablet !0;!0<<
2013-02-26 19:44:49,306 [tabletserver.TabletServer] INFO : Root tablet loaded: !0;!0<<

2013-02-26 19:45:18,254 [tabletserver.Tablet] DEBUG: Starting MajC !0;!0<< (NORMAL) [/root_tablet/A00021zc.rf, /root_tablet/F000221z.rf] --> /root_tablet/A0002529.rf_tmp  []
2013-02-26 19:45:18,371 [tabletserver.Tablet] TABLET_HIST: !0;!0<< MajC [/root_tablet/A00021zc.rf, /root_tablet/F000221z.rf] --> /root_tablet/A0002529.rf

2013-02-26 19:45:21,005 [tabletserver.Tablet] ERROR: Data file in !METADATA differ from in memory data !0;!0<<  
[
/root_tablet/A0002529.rf, 
/root_tablet/F00025hv.rf, 
/root_tablet/F000263t.rf
]  [
/root_tablet/A00021zc.rf, 
/root_tablet/F000221z.rf, 
/root_tablet/F00025hv.rf, 
/root_tablet/F000263t.rf
]

The unload/load between the servers was clean. Location states were properly made in zookeeper.