HBase
  1. HBase
  2. HBASE-6060

Regions's in OPENING state from failed regionservers takes a long time to recover

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.95.0
    • Component/s: master, regionserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      we have seen a pattern in tests, that the regions are stuck in OPENING state for a very long time when the region server who is opening the region fails. My understanding of the process:

      • master calls rs to open the region. If rs is offline, a new plan is generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), HMaster.assign()
      • RegionServer, starts opening a region, changes the state in znode. But that znode is not ephemeral. (see ZkAssign)
      • Rs transitions zk node from OFFLINE to OPENING. See OpenRegionHandler.process()
      • rs then opens the region, and changes znode from OPENING to OPENED
      • when rs is killed between OPENING and OPENED states, then zk shows OPENING state, and the master just waits for rs to change the region state, but since rs is down, that wont happen.
      • There is a AssignmentManager.TimeoutMonitor, which does exactly guard against these kind of conditions. It periodically checks (every 10 sec by default) the regions in transition to see whether they timedout (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, which explains what you and I are seeing.
      • ServerShutdownHandler in Master does not reassign regions in OPENING state, although it handles other states.

      Lowering that threshold from the configuration is one option, but still I think we can do better.

      Will investigate more.

      1. trunk-6060_v3.3.patch
        22 kB
        Jimmy Xiang
      2. trunk-6060_v2.patch
        7 kB
        Jimmy Xiang
      3. trunk-6060.patch
        4 kB
        Jimmy Xiang
      4. HBASE-6060_latest.patch
        59 kB
        ramkrishna.s.vasudevan
      5. HBASE-6060_latest.patch
        59 kB
        ramkrishna.s.vasudevan
      6. HBASE-6060_latest.patch
        57 kB
        ramkrishna.s.vasudevan
      7. HBASE-6060_trunk_5.patch
        22 kB
        rajeshbabu
      8. HBASE-6060-trunk_4.patch
        14 kB
        rajeshbabu
      9. 6060_suggestion_toassign_rs_wentdown_beforerequest.patch
        10 kB
        rajeshbabu
      10. 6060_alternative_suggestion.txt
        2 kB
        stack
      11. 6060_suggestion2_based_off_v3.patch
        30 kB
        stack
      12. 6060_suggestion_based_off_v3.patch
        30 kB
        stack
      13. 6060-trunk_3.patch
        26 kB
        rajeshbabu
      14. HBASE-6060-92.patch
        14 kB
        rajeshbabu
      15. 6060-trunk_2.patch
        27 kB
        rajeshbabu
      16. 6060-trunk.patch
        25 kB
        ramkrishna.s.vasudevan
      17. 6060-94-v4_1.patch
        28 kB
        ramkrishna.s.vasudevan
      18. 6060-trunk.patch
        25 kB
        ramkrishna.s.vasudevan
      19. 6060-94-v4_1.patch
        28 kB
        ramkrishna.s.vasudevan
      20. 6060-94-v4.patch
        28 kB
        Ted Yu
      21. 6060-94-v3.patch
        27 kB
        Ted Yu
      22. HBASE-6060-94.patch
        26 kB
        rajeshbabu

        Issue Links

          Activity

          Lars Hofhansl made changes -
          Link This issue duplicates HBASE-7724 [ HBASE-7724 ]
          stack made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          stack made changes -
          Fix Version/s 0.98.0 [ 12323143 ]
          stack made changes -
          Fix Version/s 0.98.0 [ 12323143 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.94.3 [ 12323144 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.94.3 [ 12323144 ]
          stack made changes -
          Fix Version/s 0.95.0 [ 12324094 ]
          Fix Version/s 0.96.0 [ 12320040 ]
          Ted Yu made changes -
          Link This issue is related to HBASE-7701 [ HBASE-7701 ]
          Sergey Shelukhin made changes -
          Affects Version/s 0.94.3 [ 12323144 ]
          Sergey Shelukhin made changes -
          Affects Version/s 0.94.3 [ 12323144 ]
          Jimmy Xiang made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Jimmy Xiang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jimmy Xiang made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Assignee rajeshbabu [ rajesh23 ] Jimmy Xiang [ jxiang ]
          Jimmy Xiang made changes -
          Attachment trunk-6060_v3.3.patch [ 12551735 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.92.3 [ 12321692 ]
          Fix Version/s 0.94.3 [ 12323144 ]
          Jimmy Xiang made changes -
          Attachment trunk-6060_v2.patch [ 12551096 ]
          Jimmy Xiang made changes -
          Attachment trunk-6060.patch [ 12551016 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.94.3 [ 12323144 ]
          Fix Version/s 0.94.2 [ 12321884 ]
          ramkrishna.s.vasudevan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          ramkrishna.s.vasudevan made changes -
          Attachment HBASE-6060_latest.patch [ 12541239 ]
          ramkrishna.s.vasudevan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          ramkrishna.s.vasudevan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          ramkrishna.s.vasudevan made changes -
          Attachment HBASE-6060_latest.patch [ 12541080 ]
          ramkrishna.s.vasudevan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          ramkrishna.s.vasudevan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          ramkrishna.s.vasudevan made changes -
          Attachment HBASE-6060_latest.patch [ 12540918 ]
          ramkrishna.s.vasudevan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          stack made changes -
          Fix Version/s 0.94.2 [ 12321884 ]
          Fix Version/s 0.94.1 [ 12320257 ]
          Jimmy Xiang made changes -
          Link This issue is related to HBASE-6272 [ HBASE-6272 ]
          Nicolas Liochon made changes -
          Link This issue is related to HBASE-5843 [ HBASE-5843 ]
          rajeshbabu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          rajeshbabu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          rajeshbabu made changes -
          Attachment HBASE-6060_trunk_5.patch [ 12531867 ]
          rajeshbabu made changes -
          Attachment HBASE-6060-trunk_4.patch [ 12531775 ]
          stack made changes -
          Link This issue relates to HBASE-6199 [ HBASE-6199 ]
          stack made changes -
          Link This issue relates to HBASE-6198 [ HBASE-6198 ]
          rajeshbabu made changes -
          stack made changes -
          Attachment 6060_alternative_suggestion.txt [ 12531489 ]
          stack made changes -
          Attachment 6060_suggestion2_based_off_v3.patch [ 12531373 ]
          stack made changes -
          Attachment 6060_suggestion_based_off_v3.patch [ 12531371 ]
          Ted Yu made changes -
          Link This issue relates to HBASE-5816 [ HBASE-5816 ]
          ramkrishna.s.vasudevan made changes -
          Assignee Enis Soztutar [ enis ] rajeshbabu [ rajesh23 ]
          rajeshbabu made changes -
          Attachment 6060-trunk_3.patch [ 12530962 ]
          rajeshbabu made changes -
          Attachment HBASE-6060-92.patch [ 12530816 ]
          rajeshbabu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          rajeshbabu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          rajeshbabu made changes -
          Attachment 6060-trunk_2.patch [ 12530693 ]
          Ted Yu made changes -
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.96.0 [ 12320040 ]
          Fix Version/s 0.94.1 [ 12320257 ]
          Fix Version/s 0.92.3 [ 12321692 ]
          ramkrishna.s.vasudevan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          ramkrishna.s.vasudevan made changes -
          Attachment 6060-trunk.patch [ 12530688 ]
          ramkrishna.s.vasudevan made changes -
          Attachment 6060-94-v4_1.patch [ 12530687 ]
          ramkrishna.s.vasudevan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          ramkrishna.s.vasudevan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          ramkrishna.s.vasudevan made changes -
          Attachment 6060-trunk.patch [ 12530682 ]
          ramkrishna.s.vasudevan made changes -
          Attachment 6060-94-v4_1.patch [ 12530681 ]
          Ted Yu made changes -
          Attachment 6060-94-v4.patch [ 12530633 ]
          Ted Yu made changes -
          Attachment 6060-94-v3.patch [ 12530599 ]
          rajeshbabu made changes -
          Attachment HBASE-6060-94.patch [ 12530561 ]
          ramkrishna.s.vasudevan made changes -
          Field Original Value New Value
          Link This issue is related to HBASE-5396 [ HBASE-5396 ]
          Enis Soztutar created issue -

            People

            • Assignee:
              Jimmy Xiang
              Reporter:
              Enis Soztutar
            • Votes:
              0 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development