HBase
  1. HBase
  2. HBASE-4470

ServerNotRunningException coming out of assignRootAndMeta kills the Master

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.90.4
    • Fix Version/s: 0.92.2, 0.94.1, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I'm surprised we still have issues like that and I didn't get a hit while googling so forgive me if there's already a jira about it.

      When the master starts it verifies the locations of root and meta before assigning them, if the server is started but not running you'll get this:

      2011-09-23 04:47:44,859 WARN org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: RemoteException connecting to RS
      org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningException: Server is not running yet
      at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1038)

      at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
      at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
      at $Proxy6.getProtocolVersion(Unknown Source)
      at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
      at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
      at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
      at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:969)
      at org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:388)
      at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:287)
      at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:484)
      at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:441)
      at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:388)
      at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:282)

      I hit that 3-4 times this week while debugging something else. The worst is that when you restart the master it sees that as a failover, but none of the regions are assigned so it takes an eternity to get back fully online.

      1. HBASE-4470-90.patch
        5 kB
        Gregory Chanan
      2. HBASE-4470-v2-90.patch
        6 kB
        Gregory Chanan
      3. HBASE-4470-v2-92_94.patch
        3 kB
        Gregory Chanan
      4. HBASE-4470-v2-trunk.patch
        3 kB
        Gregory Chanan

        Issue Links

          Activity

          Jean-Daniel Cryans created issue -
          Jean-Daniel Cryans made changes -
          Field Original Value New Value
          Priority Blocker [ 1 ] Critical [ 2 ]
          stack made changes -
          Fix Version/s 0.90.6 [ 12319200 ]
          Fix Version/s 0.90.5 [ 12317145 ]
          ramkrishna.s.vasudevan made changes -
          Fix Version/s 0.90.7 [ 12319481 ]
          Fix Version/s 0.90.6 [ 12319200 ]
          Jonathan Hsieh made changes -
          Link This issue relates to HBASE-5883 [ HBASE-5883 ]
          Gregory Chanan made changes -
          Link This issue is duplicated by HBASE-6151 [ HBASE-6151 ]
          Gregory Chanan made changes -
          Assignee Gregory Chanan [ gchanan ]
          Gregory Chanan made changes -
          Attachment HBASE-4470-90.patch [ 12536913 ]
          Gregory Chanan made changes -
          Attachment HBASE-4470-v2-90.patch [ 12537244 ]
          Gregory Chanan made changes -
          Attachment HBASE-4470-v2-92_94.patch [ 12537245 ]
          Gregory Chanan made changes -
          Attachment HBASE-4470-v2-trunk.patch [ 12537246 ]
          Gregory Chanan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jonathan Hsieh made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.92.2 [ 12319888 ]
          Fix Version/s 0.96.0 [ 12320040 ]
          Fix Version/s 0.94.1 [ 12320257 ]
          Resolution Fixed [ 1 ]
          Lars Hofhansl made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          stack made changes -
          Fix Version/s 0.95.0 [ 12324094 ]
          Fix Version/s 0.90.7 [ 12319481 ]
          Fix Version/s 0.92.2 [ 12319888 ]
          Fix Version/s 0.96.0 [ 12320040 ]
          Fix Version/s 0.94.1 [ 12320257 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.94.1 [ 12320257 ]
          stack made changes -
          Fix Version/s 0.92.2 [ 12319888 ]

            People

            • Assignee:
              Gregory Chanan
              Reporter:
              Jean-Daniel Cryans
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development