HBase
  1. HBase
  2. HBASE-5356

region_mover.rb can hang if table region it belongs to is deleted.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.90.3, 0.92.0, 0.94.0
    • Fix Version/s: 0.98.0, 0.96.2, 0.99.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I was testing the region_mover.rb script on a loaded hbase and noticed that it can hang (thus hanging graceful shutdown) if a region that it is attempting to move gets deleted (by a table delete operation).

      Here's the start of the relevent stack dump

      12/02/08 13:27:13 WARN client.HConnectionManager$HConnectionImplementation: Encountered problems when prefetch META table:
      org.apache.hadoop.hbase.TableNotFoundException: Cannot find row in .META. for table: TestLoadAndVerify_1328735001040, row=TestLoadAnd\
      Verify_1328735001040,yC^P\xD7\x945\xD4,99999999999999
              at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:136)
              at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:95)
              at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:64\
      9)
              at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:703\
      )
              at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:594)
              at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:565)
              at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:416)
              at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
              at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:63)
              at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.\
      java:1018)
              at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1104)
              at org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1027)
              at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:535)
              at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:525)
              at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:380)
              at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:58)
              at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:137)
              at usr.lib.hbase.bin.region_mover.method__7$RUBY$isSuccessfulScan(/usr/lib/hbase/bin/region_mover.rb:133)
              at usr$lib$hbase$bin$region_mover#method__7$RUBY$isSuccessfulScan.call(usr$lib$hbase$bin$region_mover#method__7$RUBY$isSucces\
      sfulScan:65535)
              at usr$lib$hbase$bin$region_mover#method__7$RUBY$isSuccessfulScan.call(usr$lib$hbase$bin$region_mover#method__7$RUBY$isSucces\
      sfulScan:65535)
              at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:171)
              at usr.lib.hbase.bin.region_mover.block_4$RUBY$__for__(/usr/lib/hbase/bin/region_mover.rb:326)
              at usr$lib$hbase$bin$region_mover#block_4$RUBY$__for__.call(usr$lib$hbase$bin$region_mover#block_4$RUBY$__for__:65535)
              at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:133)
              at org.jruby.runtime.BlockBody.call(BlockBody.java:73)
              at org.jruby.runtime.Block.call(Block.java:89)
              at org.jruby.RubyProc.call(RubyProc.java:268)
              at org.jruby.RubyProc.call(RubyProc.java:228)
              at org.jruby.RubyProc$i$0$0$call.call(RubyProc$i$0$0$call.gen:65535)
              at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:209)
              at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:205)
              at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:137)
              at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
              at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:103)
              at org.jruby.ast.WhileNode.interpret(WhileNode.java:131)
              at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:103)
              at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
              at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
              at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:169)
              at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:171)
              at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:272)
              at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:114)
              at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:123)
              at usr.lib.hbase.bin.region_mover.chained_26_rescue_4$RUBY$SYNTHETICunloadRegions(/usr/lib/hbase/bin/region_mover.rb:319)
              at usr.lib.hbase.bin.region_mover.method__25$RUBY$unloadRegions(/usr/lib/hbase/bin/region_mover.rb:313)
              at usr$lib$hbase$bin$region_mover#method__25$RUBY$unloadRegions.call(usr$lib$hbase$bin$region_mover#method__25$RUBY$unloadRegions:65535)
              at usr$lib$hbase$bin$region_mover#method__25$RUBY$unloadRegions.call(usr$lib$hbase$bin$region_mover#method__25$RUBY$unloadRegions:65535)
              at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:302)
              at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:173)
              at usr.lib.hbase.bin.region_mover.__file__(/usr/lib/hbase/bin/region_mover.rb:430)
              at usr.lib.hbase.bin.region_mover.load(/usr/lib/hbase/bin/region_mover.rb)
              at org.jruby.Ruby.runScript(Ruby.java:670)
              at org.jruby.Ruby.runNormally(Ruby.java:574)
              at org.jruby.Ruby.runFromMain(Ruby.java:423)
              at org.jruby.Main.doRunFromMain(Main.java:278)
              at org.jruby.Main.internalRun(Main.java:198)
              at org.jruby.Main.run(Main.java:164)
              at org.jruby.Main.run(Main.java:148)
              at org.jruby.Main.main(Main.java:128)
      

        Issue Links

          Activity

            People

            • Assignee:
              Jimmy Xiang
              Reporter:
              Jonathan Hsieh
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development