HBase
  1. HBase
  2. HBASE-869

On split, if failure updating of .META., table subsequently broke

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 0.90.0
    • Component/s: None
    • Labels:
      None

      Description

      On pset cluster – running 0.2.0 – I saw the following:

      + Deadlock on server carrying .META. made the .META. table inaccessible (deadlock has been fixed)
      + Out on a regionserver, we split; two new daughters are created and parent region is closed.
      + Regionserver fails to update the .META. with change in parent state and addition of two new daughter regions

      Restarting the server carrying .META. got us over the deadlock but subsequently, the parent region is no longer online nor its replacements.

      Attempting restart of regionserver to see if parent will come back on line (since it was not 'offlined' in .META. should come back on line again and resplit). Ugly will be the fact that the filesystem has some trash in it – the new daughter regions.

      To consider: Do not close the parent until the .META. has been successfully updated. Also, if .META. update fails, remove daughter regions.

        Issue Links

          Activity

          Hide
          stack added a comment -

          Its fixed by HBASE-2461, which will roll back reopening parent if fault. If the parent edit goes in, then the fixup in basescanner would have taken it from there. So fixed in current TRUNK. Resolving (We'll need to come up w/ a replacement for basescanner fixup when new master goes in and no basescanner). Marking duplicate.

          Show
          stack added a comment - Its fixed by HBASE-2461 , which will roll back reopening parent if fault. If the parent edit goes in, then the fixup in basescanner would have taken it from there. So fixed in current TRUNK. Resolving (We'll need to come up w/ a replacement for basescanner fixup when new master goes in and no basescanner). Marking duplicate.
          Hide
          Jean-Daniel Cryans added a comment -

          Stack, isn't this fixed with your new split code?

          Show
          Jean-Daniel Cryans added a comment - Stack, isn't this fixed with your new split code?
          Hide
          stack added a comment -

          Should be fixed as part of the master 0.21 rewrite

          Show
          stack added a comment - Should be fixed as part of the master 0.21 rewrite
          Hide
          stack added a comment -

          This is a 0.2.0 issue and the provocation has been fixed. This could still happen but rare event I'd say. Will leave it open but move it out of 0.20.0.

          Show
          stack added a comment - This is a 0.2.0 issue and the provocation has been fixed. This could still happen but rare event I'd say. Will leave it open but move it out of 0.20.0.
          Hide
          Andrew Purtell added a comment -

          Assigned to stack at his request.

          Show
          Andrew Purtell added a comment - Assigned to stack at his request.

            People

            • Assignee:
              stack
              Reporter:
              stack
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development