Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-10809

HBaseAdmin#deleteTable fails when META region happen to move around same time

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.98.0, 0.96.1, 0.99.0
    • 0.98.1, 0.99.0, 0.96.3
    • None
    • None
    • Reviewed

    Description

      The issue is that in the retry loop, we never refetch the latest meta location. So if meta starts to move right after the function get Meta server location, the delete table eventually may fail with

      org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online

      Below is the stack trace:

      2014-01-31 04:02:41,943|beaver.machine|INFO|Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException): org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online
      2014-01-31 04:02:41,943|beaver.machine|INFO|at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2585)
      2014-01-31 04:02:41,943|beaver.machine|INFO|at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3952)
      2014-01-31 04:02:41,944|beaver.machine|INFO|at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:2977)
      2014-01-31 04:02:41,944|beaver.machine|INFO|at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26929)
      2014-01-31 04:02:41,944|beaver.machine|INFO|at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
      2014-01-31 04:02:41,944|beaver.machine|INFO|at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
      2014-01-31 04:02:41,945|beaver.machine|INFO|
      2014-01-31 04:02:41,945|beaver.machine|INFO|at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1449)
      2014-01-31 04:02:41,945|beaver.machine|INFO|at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1653)
      2014-01-31 04:02:41,946|beaver.machine|INFO|at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1711)
      2014-01-31 04:02:41,946|beaver.machine|INFO|at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:27332)
      2014-01-31 04:02:41,946|beaver.machine|INFO|at org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:648)
      

      Attachments

        1. hbase-10809.patch
          2 kB
          Michael Stack
        2. hbase-10809.patch
          2 kB
          Jeffrey Zhong
        3. hbase-10809-v1.patch
          4 kB
          Jeffrey Zhong

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jeffreyz Jeffrey Zhong
            jeffreyz Jeffrey Zhong
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment