Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-2361

droptable created infinite METADATA scan loop

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.5.1
    • Fix Version/s: 1.5.1, 1.6.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      accumulo-1.5.0 with sqrrl extensions.

      Description

      Working with John Vines on this one...

      Setup: Created a couple tables, added some data, then dropped them. The drop hangs and !METADATA (which has ~400 entries) is scanned in what looks like an infinite loop.

      The table being dropped loks like this in !METADATA:

      root@sqrrl> scan -b 3 -e 5 -t !METADATA
      4;\x00\x00\x06f srv:dir []    /t-00000b6
      4;\x00\x00\x06f srv:lock []    tservers/10.10.1.209:9997/zlock-0000000000$144274cd317000b
      4;\x00\x00\x06f srv:time []    M0
      4;\x00\x00\x06f ~tab:~pr []    \x00
      4;\x00\x00\x0C\xCC loc:144274cd3170008 []    10.10.1.107:9997
      4;\x00\x00\x0C\xCC srv:dir []    /t-00000bj
      4;\x00\x00\x0C\xCC srv:lock []    tservers/10.10.1.209:9997/zlock-0000000000$144274cd317000b
      4;\x00\x00\x0C\xCC srv:time []    M0
      4;\x00\x00\x0C\xCC ~tab:~pr []    \x01\x00\x00\x06f
      4;\x00\x00\x133 srv:dir []    /t-000002h
      4;\x00\x00\x133 srv:lock []    tservers/10.10.1.209:9997/zlock-0000000000$144274cd317000b
      4;\x00\x00\x133 srv:time []    M0
      4;\x00\x00\x133 ~tab:~pr []    \x01\x00\x00\x0C\xCC
      

      We think this may be the relevant message in the master debug logs:

      2014-02-12 19:13:31,397 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
      2014-02-12 19:13:31,459 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
      2014-02-12 19:13:31,524 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
      2014-02-12 19:13:31,588 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
      2014-02-12 19:13:31,662 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
      2014-02-12 19:13:31,725 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
      2014-02-12 19:13:31,788 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
      2014-02-12 19:13:31,854 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
      2014-02-12 19:13:31,917 [tableOps.CleanUp] DEBUG: Still waiting for table to be deleted: 4 saw inconsistencynull 4;^@^@^L?;^@^@^Ff
      ...etc
      

      Graceful accumulo reboot hangs.

      Hard reboot of everything (control-c'd) clears the problem.

        Attachments

        1. TableCreationHammerIT.java
          7 kB
          Eric Newton
        2. AccumuloHammerTableCreation.java
          5 kB
          John Vines
        3. AccumuloHammerTableCreation.java
          6 kB
          John Vines
        4. short.debug.log
          21 kB
          Chris McCubbin
        5. Screen Shot 2014-02-12 at 2.27.11 PM.png
          86 kB
          Chris McCubbin
        6. Screen Shot 2014-02-12 at 2.28.09 PM.png
          90 kB
          Chris McCubbin
        7. masterJstack3.txt
          43 kB
          Chris McCubbin
        8. masterJstack2.txt
          39 kB
          Chris McCubbin
        9. masterJstack.txt
          42 kB
          Chris McCubbin
        10. jstack2.txt
          165 kB
          Chris McCubbin
        11. jstack1.txt
          153 kB
          Chris McCubbin
        12. jstack3.txt
          188 kB
          Chris McCubbin

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                cmccubbin Chris McCubbin
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: