[HBASE-7739] [snapshot 130201 merge] Fix TestSnapshotFromMaster failure

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.95.0
• Hadoop Flags: Reviewed
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 02/Feb/13 00:00
• Updated: 23/Sep/13 18:30
• Resolved: 02/Feb/13 18:43

Description

Looking at test output from TestSnapshotFromMaster:

2013-02-01 14:10:49,234 INFO  [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] regionserver.SplitTransaction(215): Starting split of region test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae.
2013-02-01 14:10:49,235 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] regionserver.SplitTransaction(886): regionserver:53794-0x13c97cfa9760002 Creating ephemeral node for e9c665350a0d6d059cad8ae03f8dadae in SPLITTING state
2013-02-01 14:10:49,236 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(265): master:53791-0x13c97cfa9760000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition
2013-02-01 14:10:49,236 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] zookeeper.ZKAssign(720): regionserver:53794-0x13c97cfa9760002 Attempting to transition node e9c665350a0d6d059cad8ae03f8dadae from RS_ZK_REGION_SPLITTING to RS_ZK_REGION_SPLITTING
2013-02-01 14:10:49,238 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] zookeeper.ZKUtil(1643): regionserver:53794-0x13c97cfa9760002 Retrieved 94 byte(s) of data from znode /hbase/region-in-transition/e9c665350a0d6d059cad8ae03f8dadae; data=region=test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae., state=RS_ZK_REGION_SPLITTING, servername=10.11.3.105,53794,1359756635266, createTime=1359756649235, payload.length=0
2013-02-01 14:10:49,238 DEBUG [hbase-am-zkevent-worker-pool-2-thread-16] zookeeper.ZKUtil(1643): master:53791-0x13c97cfa9760000 Retrieved 94 byte(s) of data from znode /hbase/region-in-transition/e9c665350a0d6d059cad8ae03f8dadae and set watcher; region=test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae., state=RS_ZK_REGION_SPLITTING, servername=10.11.3.105,53794,1359756635266, createTime=1359756649235, payload.length=0
2013-02-01 14:10:49,239 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(265): master:53791-0x13c97cfa9760000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/e9c665350a0d6d059cad8ae03f8dadae
2013-02-01 14:10:49,240 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] zookeeper.ZKAssign(782): regionserver:53794-0x13c97cfa9760002 Successfully transitioned node e9c665350a0d6d059cad8ae03f8dadae from RS_ZK_REGION_SPLITTING to RS_ZK_REGION_SPLITTING
2013-02-01 14:10:49,240 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] zookeeper.ZKAssign(720): regionserver:53794-0x13c97cfa9760002 Attempting to transition node e9c665350a0d6d059cad8ae03f8dadae from RS_ZK_REGION_SPLITTING to RS_ZK_REGION_SPLITTING
2013-02-01 14:10:49,240 DEBUG [hbase-am-zkevent-worker-pool-2-thread-17] zookeeper.ZKUtil(1643): master:53791-0x13c97cfa9760000 Retrieved 94 byte(s) of data from znode /hbase/region-in-transition/e9c665350a0d6d059cad8ae03f8dadae and set watcher; region=test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae., state=RS_ZK_REGION_SPLITTING, servername=10.11.3.105,53794,1359756635266, createTime=1359756649238, payload.length=0
2013-02-01 14:10:49,241 DEBUG [hbase-am-zkevent-worker-pool-2-thread-17] master.AssignmentManager(641): Handling transition=RS_ZK_REGION_SPLITTING, server=10.11.3.105,53794,1359756635266, region=e9c665350a0d6d059cad8ae03f8dadae, current state from region state map =null
2013-02-01 14:10:49,241 INFO  [hbase-am-zkevent-worker-pool-2-thread-17] master.RegionStates(242): Region {NAME => 'test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae.', STARTKEY => '', ENDKEY => '', ENCODED => e9c665350a0d6d059cad8ae03f8dadae,} transitioned from {test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae. state=OPEN, ts=1359756647443, server=10.11.3.105,53794,1359756635266} to {test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae. state=SPLITTING, ts=1359756649241, server=10.11.3.105,53794,1359756635266}
2013-02-01 14:10:49,241 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] zookeeper.ZKUtil(1643): regionserver:53794-0x13c97cfa9760002 Retrieved 94 byte(s) of data from znode /hbase/region-in-transition/e9c665350a0d6d059cad8ae03f8dadae; data=region=test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae., state=RS_ZK_REGION_SPLITTING, servername=10.11.3.105,53794,1359756635266, createTime=1359756649238, payload.length=0
2013-02-01 14:10:49,243 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(265): master:53791-0x13c97cfa9760000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/e9c665350a0d6d059cad8ae03f8dadae
2013-02-01 14:10:49,243 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] zookeeper.ZKAssign(782): regionserver:53794-0x13c97cfa9760002 Successfully transitioned node e9c665350a0d6d059cad8ae03f8dadae from RS_ZK_REGION_SPLITTING to RS_ZK_REGION_SPLITTING
2013-02-01 14:10:49,244 DEBUG [hbase-am-zkevent-worker-pool-2-thread-18] zookeeper.ZKUtil(1643): master:53791-0x13c97cfa9760000 Retrieved 94 byte(s) of data from znode /hbase/region-in-transition/e9c665350a0d6d059cad8ae03f8dadae and set watcher; region=test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae., state=RS_ZK_REGION_SPLITTING, servername=10.11.3.105,53794,1359756635266, createTime=1359756649242, payload.length=0
2013-02-01 14:10:49,245 DEBUG [hbase-am-zkevent-worker-pool-2-thread-18] master.AssignmentManager(641): Handling transition=RS_ZK_REGION_SPLITTING, server=10.11.3.105,53794,1359756635266, region=e9c665350a0d6d059cad8ae03f8dadae, current state from region state map =null
2013-02-01 14:10:49,245 INFO  [hbase-am-zkevent-worker-pool-2-thread-18] master.RegionStates(242): Region {NAME => 'test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae.', STARTKEY => '', ENDKEY => '', ENCODED => e9c665350a0d6d059cad8ae03f8dadae,} transitioned from {test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae. state=SPLITTING, ts=1359756649241, server=10.11.3.105,53794,1359756635266} to {test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae. state=SPLITTING, ts=1359756649245, server=10.11.3.105,53794,1359756635266}
2013-02-01 14:10:49,246 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] regionserver.HRegion(1005): Closing test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae.: disabling compactions & flushes
2013-02-01 14:10:49,246 WARN  [IPC Server handler 1 on 53794] regionserver.HRegion(2133): Failed getting lock in batch put, row=hox
org.apache.hadoop.hbase.NotServingRegionException: test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae. is closing
  at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:5319)
  at org.apache.hadoop.hbase.regionserver.HRegion.internalObtainRowLock(HRegion.java:3181)
  at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:3269)
  at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2130)
  at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2004)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:3862)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3307)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at org.apache.hadoop.hbase.ipc.ProtobufRpcServerEngine$Server.call(ProtobufRpcServerEngine.java:197)
  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1786)
2013-02-01 14:10:49,257 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] regionserver.HRegion(1027): Updates disabled for region test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae.

I think we should replace IncreasingToUpperBoundRegionSplitPolicy in TestSnapshotFromMaster.