Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-7698

race between RS shutdown thread and openregionhandler causes region to get stuck

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.6, 0.95.0
    • Fix Version/s: 0.94.5, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      2013-01-22 17:59:03,237 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5984cf08] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(186): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5984cf08
      ...
      2013-01-22 17:59:03,411 DEBUG [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] regionserver.HRegion(1001): Closing IntegrationTestRebalanceAndKillServersTargeted,66666660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb.: disabling compactions & flushes
      2013-01-22 17:59:03,411 DEBUG [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] regionserver.HRegion(1023): Updates disabled for region IntegrationTestRebalanceAndKillServersTargeted,66666660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb.
      2013-01-22 17:59:03,415 ERROR [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] executor.EventHandler(205): Caught throwable while processing event M_RS_OPEN_REGION
      java.io.IOException: java.io.IOException: java.io.IOException: Filesystem closed
      at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1058)
      at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:974)
      at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:945)
      at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.cleanupFailedOpen(OpenRegionHandler.java:459)
      at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:143)
      at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:680)

      tryTransitionFromOpeningToFailedOpen or transitionToOpened below is never called and region can get stuck.
      As an added benefit, the meta is already written by that time.

        Attachments

        1. HBASE-7698.patch
          2 kB
          ramkrishna.s.vasudevan
        2. HBASE-7698_withtestcase.patch
          3 kB
          ramkrishna.s.vasudevan
        3. HBASE-7698_withtestcase_1.patch
          4 kB
          ramkrishna.s.vasudevan
        4. HBASE-7698_withtestcase_1.patch
          3 kB
          ramkrishna.s.vasudevan
        5. HBASE-7698_0.94.patch
          4 kB
          ramkrishna.s.vasudevan
        6. HBASE-7698_trunk_final.patch
          3 kB
          ramkrishna.s.vasudevan

          Issue Links

            Activity

              People

              • Assignee:
                ram_krish ramkrishna.s.vasudevan
                Reporter:
                sershe Sergey Shelukhin
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: