HBase
  1. HBase
  2. HBASE-3152

Disabling table stuck on regions-in-transition

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.90.0
    • Fix Version/s: None
    • Component/s: master
    • Labels:
      None

      Description

      after stress testing with only ICV and trying to disable table hbase run into regions-in-transition loop. I have logs and threads dumps. This region is on db2a regionserver, more logs as attachments

      2010-10-26 09:32:00,838 INFO org.apache.hadoop.hbase.master.AssignmentManager: Waiting on NGolden_AC,,1287997478278.85b04e1097c676ef52b16d49305e0ab0. state=PENDING_CLOSE, ts=1288078080792 to clear regions-in-transition
      2010-10-26 09:32:01,838 INFO org.apache.hadoop.hbase.master.AssignmentManager: Waiting on NGolden_AC,,1287997478278.85b04e1097c676ef52b16d49305e0ab0. state=PENDING_CLOSE, ts=1288078080792 to clear regions-in-transition
      2010-10-26 09:32:02,838 INFO org.apache.hadoop.hbase.master.AssignmentManager: Waiting on NGolden_AC,,1287997478278.85b04e1097c676ef52b16d49305e0ab0. state=PENDING_CLOSE, ts=1288078080792 to clear regions-in-transition
      2010-10-26 09:32:02,842 INFO org.apache.hadoop.hbase.master.ServerManager: regionservers=2, averageload=1803
      2010-10-26 09:32:03,104 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: NGolden_AC,,1287997478278.85b04e1097c676ef52b16d49305e0ab0. state=PENDING_CLOSE, ts=1288078080792
      2010-10-26 09:32:03,104 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been PENDING_CLOSE or CLOSING for too long, running forced unassign again on region=NGolden_AC,,1287997478278.85b04e1097c676ef52b16d49305e0ab0.
      2010-10-26 09:32:03,104 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12be24df47b0000 Deleting existing unassigned node for 85b04e1097c676ef52b16d49305e0ab0 that is in expected state RS_ZK_REGION_CLOSING
      2010-10-26 09:32:03,104 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12be24df47b0000 Attempting to delete unassigned node in RS_ZK_REGION_CLOSING state but node is in RS_ZK_REGION_CLOSED state
      2010-10-26 09:32:03,838 INFO org.apache.hadoop.hbase.master.AssignmentManager: Waiting on NGolden_AC,,1287997478278.85b04e1097c676ef52b16d49305e0ab0. state=PENDING_CLOSE, ts=1288078080792 to clear regions-in-transition
      2010-10-26 09:32:04,838 INFO org.apache.hadoop.hbase.master.AssignmentManager: Waiting on NGolden_AC,,1287997478278.85b04e1097c676ef52b16d49305e0ab0. state=PENDING_CLOSE, ts=1288078080792 to clear regions-in-transition
      2010-10-26 09:32:05,838 INFO org.apache.hadoop.hbase.master.AssignmentManager: Waiting on NGolden_AC,,1287997478278.85b04e1097c676ef52b16d49305e0ab0. state=PENDING_CLOSE, ts=1288078080792 to clear regions-in-transition

      1. hbase-hbase-master-db2a.goldenline.pl.log.gz
        83 kB
        Sebastian Bauer
      2. hbase-hbase-master-db2a.goldenline.pl.out.gz
        4 kB
        Sebastian Bauer
      3. hbase-hbase-regionserver-db2a.goldenline.pl.log.gz
        1.10 MB
        Sebastian Bauer
      4. hbase-hbase-regionserver-db2a.goldenline.pl.out.gz
        5 kB
        Sebastian Bauer
      5. hbase-hbase-regionserver-db2b.goldenline.pl.log.gz
        4.01 MB
        Sebastian Bauer
      6. hbase-hbase-regionserver-db2b.goldenline.pl.out.gz
        5 kB
        Sebastian Bauer

        Issue Links

          Activity

          Hide
          stack added a comment -

          Moving out of 0.90. This looks like it should have been fixed by HBASE-3147.

          Show
          stack added a comment - Moving out of 0.90. This looks like it should have been fixed by HBASE-3147 .
          Hide
          stack added a comment -

          Moving out of 0.92.0. Pull it back in if you think different.

          Show
          stack added a comment - Moving out of 0.92.0. Pull it back in if you think different.
          Hide
          Harsh J added a comment -

          This shouldn't happen anymore - we prevent racing between the two IIUC.

          Also see HBASE-6537 for 0.94, and Stack's comment above.

          Show
          Harsh J added a comment - This shouldn't happen anymore - we prevent racing between the two IIUC. Also see HBASE-6537 for 0.94, and Stack's comment above.

            People

            • Assignee:
              stack
              Reporter:
              Sebastian Bauer
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development