Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-6060

Regions's in OPENING state from failed regionservers takes a long time to recover

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.95.0
    • Component/s: master, regionserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      we have seen a pattern in tests, that the regions are stuck in OPENING state for a very long time when the region server who is opening the region fails. My understanding of the process:

      • master calls rs to open the region. If rs is offline, a new plan is generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), HMaster.assign()
      • RegionServer, starts opening a region, changes the state in znode. But that znode is not ephemeral. (see ZkAssign)
      • Rs transitions zk node from OFFLINE to OPENING. See OpenRegionHandler.process()
      • rs then opens the region, and changes znode from OPENING to OPENED
      • when rs is killed between OPENING and OPENED states, then zk shows OPENING state, and the master just waits for rs to change the region state, but since rs is down, that wont happen.
      • There is a AssignmentManager.TimeoutMonitor, which does exactly guard against these kind of conditions. It periodically checks (every 10 sec by default) the regions in transition to see whether they timedout (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, which explains what you and I are seeing.
      • ServerShutdownHandler in Master does not reassign regions in OPENING state, although it handles other states.

      Lowering that threshold from the configuration is one option, but still I think we can do better.

      Will investigate more.

        Attachments

        1. 6060_alternative_suggestion.txt
          2 kB
          Michael Stack
        2. 6060_suggestion_based_off_v3.patch
          30 kB
          Michael Stack
        3. 6060_suggestion_toassign_rs_wentdown_beforerequest.patch
          10 kB
          rajeshbabu
        4. 6060_suggestion2_based_off_v3.patch
          30 kB
          Michael Stack
        5. 6060-94-v3.patch
          27 kB
          Ted Yu
        6. 6060-94-v4_1.patch
          28 kB
          ramkrishna.s.vasudevan
        7. 6060-94-v4_1.patch
          28 kB
          ramkrishna.s.vasudevan
        8. 6060-94-v4.patch
          28 kB
          Ted Yu
        9. 6060-trunk_2.patch
          27 kB
          rajeshbabu
        10. 6060-trunk_3.patch
          26 kB
          rajeshbabu
        11. 6060-trunk.patch
          25 kB
          ramkrishna.s.vasudevan
        12. 6060-trunk.patch
          25 kB
          ramkrishna.s.vasudevan
        13. HBASE-6060_latest.patch
          59 kB
          ramkrishna.s.vasudevan
        14. HBASE-6060_latest.patch
          59 kB
          ramkrishna.s.vasudevan
        15. HBASE-6060_latest.patch
          57 kB
          ramkrishna.s.vasudevan
        16. HBASE-6060_trunk_5.patch
          22 kB
          rajeshbabu
        17. HBASE-6060-92.patch
          14 kB
          rajeshbabu
        18. HBASE-6060-94.patch
          26 kB
          rajeshbabu
        19. HBASE-6060-trunk_4.patch
          14 kB
          rajeshbabu
        20. trunk-6060_v2.patch
          7 kB
          Jimmy Xiang
        21. trunk-6060_v3.3.patch
          22 kB
          Jimmy Xiang
        22. trunk-6060.patch
          4 kB
          Jimmy Xiang

          Issue Links

            Activity

              People

              • Assignee:
                jxiang Jimmy Xiang
                Reporter:
                enis Enis Soztutar
              • Votes:
                0 Vote for this issue
                Watchers:
                21 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: