HBase
  1. HBase
  2. HBASE-10085

Some regions aren't re-assigned after a cluster restarts

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.96.1
    • Fix Version/s: 0.98.0, 0.96.1
    • Component/s: Region Assignment
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We see this issue happened in a cluster restart:

      1) when shutdown a cluster, some regions are in offline state because no Region servers are available(stop RS and then Master)
      2) When the cluster restarts, the offlined regions are forced to be offline again and SSH skip re-assigning them by function AM.processServerShutdown as shown below.

      2013-12-03 10:41:56,686 INFO  [master:h2-ubuntu12-sec-1386048659-hbase-8:60000] master.AssignmentManager: Processing 873dbd8c269f44d0aefb0f66c5b53537 in state: M_ZK_REGION_OFFLINE
      2013-12-03 10:41:56,686 DEBUG [master:h2-ubuntu12-sec-1386048659-hbase-8:60000] master.AssignmentManager: RIT 873dbd8c269f44d0aefb0f66c5b53537 in state=M_ZK_REGION_OFFLINE was on deadserver; forcing offline
      ...
      2013-12-03 10:41:56,739 DEBUG [AM.-pool1-t8] master.AssignmentManager: Force region state offline {873dbd8c269f44d0aefb0f66c5b53537 state=OFFLINE, ts=1386067316737, server=h2-ubuntu12-sec-1386048659-hbase-6.cs1cloud.internal,60020,1386066968696}
      ...
      2013-12-03 10:41:57,223 WARN  [MASTER_SERVER_OPERATIONS-h2-ubuntu12-sec-1386048659-hbase-8:60000-3] master.RegionStates: THIS SHOULD NOT HAPPEN: unexpected {873dbd8c269f44d0aefb0f66c5b53537 state=OFFLINE, ts=1386067316737, server=h2-ubuntu12-sec-1386048659-hbase-6.cs1cloud.internal,60020,1386066968696} 
      
      1. hbase-10085.patch
        7 kB
        Jeffrey Zhong

        Issue Links

          Activity

          Jeffrey Zhong created issue -
          Jeffrey Zhong made changes -
          Field Original Value New Value
          Affects Version/s 0.96.1 [ 12324961 ]
          Affects Version/s 0.96.0 [ 12324822 ]
          Jeffrey Zhong made changes -
          Attachment hbase-10085.patch [ 12617100 ]
          Jeffrey Zhong made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jonathan Hsieh made changes -
          Summary Some regions aren't re-assigned after a mater restarts Some regions aren't re-assigned after a master restarts
          Jimmy Xiang made changes -
          Link This issue is depended upon by HBASE-10090 [ HBASE-10090 ]
          Jeffrey Zhong made changes -
          Attachment hbase-10085.patch [ 12617308 ]
          Jeffrey Zhong made changes -
          Attachment hbase-10085.patch [ 12617100 ]
          Jeffrey Zhong made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Jeffrey Zhong made changes -
          Summary Some regions aren't re-assigned after a master restarts Some regions aren't re-assigned after a cluster restarts
          stack made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Jeffrey Zhong
              Reporter:
              Jeffrey Zhong
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development