Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-17287

Master becomes a zombie if filesystem object closes

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.0, 1.3.1, 1.1.10, 1.2.6, 2.0.0
    • Component/s: master
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      If filesystem is not available during log split, abort master server.

      Description

      We have seen an issue whereby if the HDFS is unstable and the HBase master's HDFS client is unable to stabilize before dfs.client.failover.max.attempts then the master's filesystem object closes. This seems to result in an HBase master which will continue to run (process and znode exists) but no meaningful work can be done (e.g. assigning meta).What we saw in our HBase master logs was:

      2016-12-01 19:19:08,192 ERROR org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler: Caught M_META_SERVER_SHUTDOWN, count=1java.io.IOException: failed log splitting for cluster-r5n12.bloomberg.com,60200,1480632863218, will retryat org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:84)at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)Caused by: java.io.IOException: Filesystem closed

        Attachments

        1. 17287.branch-1.1.v4.txt
          6 kB
          Ted Yu
        2. 17287.branch-1.v3.txt
          0.7 kB
          Ted Yu
        3. 17287.branch-1.v4.txt
          6 kB
          Ted Yu
        4. 17287.master.v2.txt
          0.8 kB
          Ted Yu
        5. 17287.master.v3.txt
          0.7 kB
          Ted Yu
        6. 17287.master.v4.txt
          4 kB
          Ted Yu
        7. 17287.master.v5.txt
          6 kB
          Ted Yu
        8. 17287.v2.txt
          0.8 kB
          Ted Yu

          Activity

            People

            • Assignee:
              yuzhihong@gmail.com Ted Yu
              Reporter:
              clayb Clay B.
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: