[ACCUMULO-2361] droptable created infinite METADATA scan loop

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.5.1
• Fix Version/s: 1.5.1, 1.6.0

Environment

accumulo-1.5.0 with sqrrl extensions.

• Assignee: Unassigned
• Reporter: Chris McCubbin
• Created: 12/Feb/14 20:58
• Updated: 23/Jan/15 21:10
• Resolved: 23/Jan/15 21:10

Description

Working with vines on this one...

Setup: Created a couple tables, added some data, then dropped them. The drop hangs and !METADATA (which has ~400 entries) is scanned in what looks like an infinite loop.

The table being dropped loks like this in !METADATA:

root@sqrrl> scan -b 3 -e 5 -t !METADATA
4;\x00\x00\x06f srv:dir []    /t-00000b6
4;\x00\x00\x06f srv:lock []    tservers/10.10.1.209:9997/zlock-0000000000$144274cd317000b
4;\x00\x00\x06f srv:time []    M0
4;\x00\x00\x06f ~tab:~pr []    \x00
4;\x00\x00\x0C\xCC loc:144274cd3170008 []    10.10.1.107:9997
4;\x00\x00\x0C\xCC srv:dir []    /t-00000bj
4;\x00\x00\x0C\xCC srv:lock []    tservers/10.10.1.209:9997/zlock-0000000000$144274cd317000b
4;\x00\x00\x0C\xCC srv:time []    M0
4;\x00\x00\x0C\xCC ~tab:~pr []    \x01\x00\x00\x06f
4;\x00\x00\x133 srv:dir []    /t-000002h
4;\x00\x00\x133 srv:lock []    tservers/10.10.1.209:9997/zlock-0000000000$144274cd317000b
4;\x00\x00\x133 srv:time []    M0
4;\x00\x00\x133 ~tab:~pr []    \x01\x00\x00\x0C\xCC

We think this may be the relevant message in the master debug logs:

2014-02-12 19:13:31,397 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
2014-02-12 19:13:31,459 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
2014-02-12 19:13:31,524 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
2014-02-12 19:13:31,588 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
2014-02-12 19:13:31,662 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
2014-02-12 19:13:31,725 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
2014-02-12 19:13:31,788 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
2014-02-12 19:13:31,854 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
2014-02-12 19:13:31,917 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
...etc

Graceful accumulo reboot hangs.

Hard reboot of everything (control-c'd) clears the problem.

Issue Links

• relates to: ACCUMULO-2489 NPE in split tablet