Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-3446

ProcessServerShutdown fails if META moves, orphaning lots of regions

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.90.0
    • Fix Version/s: 0.92.0
    • Component/s: master
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Makes catalog/* classes retry: e.g. MetaEditor, MetaReader and CatalogTracker. Previously they would try once and unless successful, fail. Retrying is courtesy of HTable instances.

      Description

      I ran a rolling restart on a 5 node cluster with lots of regions, and afterwards had LOTS of regions left orphaned. The issue appears to be that ProcessServerShutdown failed because the server hosting META was restarted around the same time as another server was being processed

        Attachments

        1. 3446.txt
          12 kB
          Michael Stack
        2. 3446-v11.txt
          63 kB
          Michael Stack
        3. 3446-v12.txt
          68 kB
          Michael Stack
        4. 3446-v13.txt
          75 kB
          Michael Stack
        5. 3446-v14.txt
          81 kB
          Michael Stack
        6. 3446v15.txt
          101 kB
          Michael Stack
        7. 3446-v2.txt
          22 kB
          Michael Stack
        8. 3446v23.txt
          174 kB
          Michael Stack
        9. 3446-v3.txt
          50 kB
          Michael Stack
        10. 3446-v4.txt
          53 kB
          Michael Stack
        11. 3446-v7.txt
          58 kB
          Michael Stack
        12. 3446-v9.txt
          60 kB
          Michael Stack

          Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. HBASE-3138 When new master joins running cluster but meta is yanked from it as processing RIT, gets unexpected state

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HBASE-3478 HBase fails to recover from failed DNS resolution of stale meta connection info

          • Major - Major loss of function.
          • Resolved

            Activity

              People

              • Assignee:
                stack Michael Stack
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: