HBase
  1. HBase
  2. HBASE-3138

When new master joins running cluster but meta is yanked from it as processing RIT, gets unexpected state

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: None
    • Fix Version/s: 0.92.3
    • Component/s: None
    • Labels:
      None

      Description

      Testing rolling restart i turned up the following condition.

      Master is joining an extant cluster and is trying to clean up RIT. Then the server hosting .META. is shutdown in the middle of it all. Deal. Here is exception.

      2010-10-21 06:45:58,592 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=sv2borg187,60020,1287643131919, region=efcd899283e96f20faa317772f52adca
      2010-10-21 06:45:58,616 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
      org.apache.hadoop.ipc.RemoteException: java.io.IOException: Server not running
          at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2198)
          at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1499)
          at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
          at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
      
          at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:749)
          at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:255)
          at $Proxy1.get(Unknown Source)
          at org.apache.hadoop.hbase.catalog.MetaReader.getRegion(MetaReader.java:286)
          at org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransition(AssignmentManager.java:250)
          at org.apache.hadoop.hbase.master.AssignmentManager.processFailover(AssignmentManager.java:209)
          at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:392)
          at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:268)
      2010-10-21 06:45:58,617 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
      

        Issue Links

          Activity

          stack created issue -
          stack made changes -
          Field Original Value New Value
          Priority Major [ 3 ] Blocker [ 1 ]
          stack made changes -
          Fix Version/s 0.92.0 [ 12314223 ]
          Fix Version/s 0.90.0 [ 12313607 ]
          stack made changes -
          Link This issue is blocked by HBASE-3446 [ HBASE-3446 ]
          stack made changes -
          Priority Blocker [ 1 ] Major [ 3 ]
          stack made changes -
          Fix Version/s 0.92.1 [ 12318551 ]
          Fix Version/s 0.92.0 [ 12314223 ]
          stack made changes -
          Fix Version/s 0.92.2 [ 12319888 ]
          Fix Version/s 0.92.1 [ 12318551 ]
          stack made changes -
          Fix Version/s 0.92.3 [ 12321692 ]
          Fix Version/s 0.92.2 [ 12319888 ]
          Andrew Purtell made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Cannot Reproduce [ 5 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              stack
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development