[HBASE-5356] region_mover.rb can hang if table region it belongs to is deleted.

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.90.3, 0.92.0, 0.94.0
• Fix Version/s: 0.98.0, 0.96.2, 0.99.0
• Hadoop Flags: Reviewed
• Assignee: Jimmy Xiang
• Reporter: Jonathan Hsieh
• Created: 08/Feb/12 22:12
• Updated: 16/Jun/15 23:39
• Resolved: 07/Feb/14 18:00

Description

I was testing the region_mover.rb script on a loaded hbase and noticed that it can hang (thus hanging graceful shutdown) if a region that it is attempting to move gets deleted (by a table delete operation).

Here's the start of the relevent stack dump

12/02/08 13:27:13 WARN client.HConnectionManager$HConnectionImplementation: Encountered problems when prefetch META table:
org.apache.hadoop.hbase.TableNotFoundException: Cannot find row in .META. for table: TestLoadAndVerify_1328735001040, row=TestLoadAnd\
Verify_1328735001040,yC^P\xD7\x945\xD4,99999999999999
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:136)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:95)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:64\
9)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:703\
)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:594)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:565)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:416)
        at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
        at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:63)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.\
java:1018)
        at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1104)
        at org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1027)
        at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:535)
        at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:525)
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:380)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:58)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:137)
        at usr.lib.hbase.bin.region_mover.method__7$RUBY$isSuccessfulScan(/usr/lib/hbase/bin/region_mover.rb:133)
        at usr$lib$hbase$bin$region_mover#method__7$RUBY$isSuccessfulScan.call(usr$lib$hbase$bin$region_mover#method__7$RUBY$isSucces\
sfulScan:65535)
        at usr$lib$hbase$bin$region_mover#method__7$RUBY$isSuccessfulScan.call(usr$lib$hbase$bin$region_mover#method__7$RUBY$isSucces\
sfulScan:65535)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:171)
        at usr.lib.hbase.bin.region_mover.block_4$RUBY$__for__(/usr/lib/hbase/bin/region_mover.rb:326)
        at usr$lib$hbase$bin$region_mover#block_4$RUBY$__for__.call(usr$lib$hbase$bin$region_mover#block_4$RUBY$__for__:65535)
        at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:133)
        at org.jruby.runtime.BlockBody.call(BlockBody.java:73)
        at org.jruby.runtime.Block.call(Block.java:89)
        at org.jruby.RubyProc.call(RubyProc.java:268)
        at org.jruby.RubyProc.call(RubyProc.java:228)
        at org.jruby.RubyProc$i$0$0$call.call(RubyProc$i$0$0$call.gen:65535)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:209)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:205)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:137)
        at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:103)
        at org.jruby.ast.WhileNode.interpret(WhileNode.java:131)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:103)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:169)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:171)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:272)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:114)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:123)
        at usr.lib.hbase.bin.region_mover.chained_26_rescue_4$RUBY$SYNTHETICunloadRegions(/usr/lib/hbase/bin/region_mover.rb:319)
        at usr.lib.hbase.bin.region_mover.method__25$RUBY$unloadRegions(/usr/lib/hbase/bin/region_mover.rb:313)
        at usr$lib$hbase$bin$region_mover#method__25$RUBY$unloadRegions.call(usr$lib$hbase$bin$region_mover#method__25$RUBY$unloadRegions:65535)
        at usr$lib$hbase$bin$region_mover#method__25$RUBY$unloadRegions.call(usr$lib$hbase$bin$region_mover#method__25$RUBY$unloadRegions:65535)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:302)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:173)
        at usr.lib.hbase.bin.region_mover.__file__(/usr/lib/hbase/bin/region_mover.rb:430)
        at usr.lib.hbase.bin.region_mover.load(/usr/lib/hbase/bin/region_mover.rb)
        at org.jruby.Ruby.runScript(Ruby.java:670)
        at org.jruby.Ruby.runNormally(Ruby.java:574)
        at org.jruby.Ruby.runFromMain(Ruby.java:423)
        at org.jruby.Main.doRunFromMain(Main.java:278)
        at org.jruby.Main.internalRun(Main.java:198)
        at org.jruby.Main.run(Main.java:164)
        at org.jruby.Main.run(Main.java:148)
        at org.jruby.Main.main(Main.java:128)

Issue Links

• is duplicated by: HBASE-8490 Region mover script can get stuck in infinite loop if UnknownRegionException