[ACCUMULO-1888] WALog entries not properly deleted after recovery during upgrade 1.5 -> 1.6

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: tserver
• Assignee: Eric C. Newton
• Reporter: Eric C. Newton
• Created: 13/Nov/13 15:24
• Updated: 13/Nov/13 16:46
• Resolved: 13/Nov/13 16:39

Description

While investigating ACCUMULO-1831, I got close consistency check failures:

2013-11-13 09:47:43,797 [tserver.Tablet] ERROR: Closed tablet 2;row_0000020000< has walog entries in !METADATA [2;row_0000020000; da20ecd9-2e3d-4165-ae49-351b56178b2f (12)]

Examining the WALog for !METADATA updates to this tablet, I see the walog entry being deleted:

MANY_MUTATIONS 4 5
1 mutations:
  2;row_0000020000
      file:hdfs://hostname:9000/accumulo/tables/2/t-0000003/F000008v.rf [system]:73 [] 863869,20001
      srv:time [system]:73 [] M0
      last:141e12559a3ffc2 [system]:73 [] localhost:9997
      log:127.0.0.1:9997/127.0.0.1+9997/da20ecd9-2e3d-4165-ae49-351b56178b2f [system]:73 [] <deleted>
      log:127.0.0.1+9997/da20ecd9-2e3d-4165-ae49-351b56178b2f
      log:localhost:9997/hdfs://hostname:9000/accumulo/wal/localhost+9997/48f53ae2-2b54-4309-88c7-003c723dcd0e [system]:73 [] <deleted>
      srv:flush [system]:73 [] 0
      srv:lock [system]:73 [] tservers/localhost:9997/zlock-0000000000$141e12559a3ffc2

The metadata table contained the key:

2;row_0000020000 log:127.0.0.1+9997/da20ecd9-2e3d-4165-ae49-351b56178b2f

But this key was used to remove it:

2;row_0000020000 log:127.0.0.1:9997/127.0.0.1+9997/da20ecd9-2e3d-4165-ae49-351b56178b2f