Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-7290 Online snapshots
  3. HBASE-7739

[snapshot 130201 merge] Fix TestSnapshotFromMaster failure

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Looking at test output from TestSnapshotFromMaster:

      2013-02-01 14:10:49,234 INFO  [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] regionserver.SplitTransaction(215): Starting split of region test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae.
      2013-02-01 14:10:49,235 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] regionserver.SplitTransaction(886): regionserver:53794-0x13c97cfa9760002 Creating ephemeral node for e9c665350a0d6d059cad8ae03f8dadae in SPLITTING state
      2013-02-01 14:10:49,236 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(265): master:53791-0x13c97cfa9760000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition
      2013-02-01 14:10:49,236 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] zookeeper.ZKAssign(720): regionserver:53794-0x13c97cfa9760002 Attempting to transition node e9c665350a0d6d059cad8ae03f8dadae from RS_ZK_REGION_SPLITTING to RS_ZK_REGION_SPLITTING
      2013-02-01 14:10:49,238 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] zookeeper.ZKUtil(1643): regionserver:53794-0x13c97cfa9760002 Retrieved 94 byte(s) of data from znode /hbase/region-in-transition/e9c665350a0d6d059cad8ae03f8dadae; data=region=test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae., state=RS_ZK_REGION_SPLITTING, servername=10.11.3.105,53794,1359756635266, createTime=1359756649235, payload.length=0
      2013-02-01 14:10:49,238 DEBUG [hbase-am-zkevent-worker-pool-2-thread-16] zookeeper.ZKUtil(1643): master:53791-0x13c97cfa9760000 Retrieved 94 byte(s) of data from znode /hbase/region-in-transition/e9c665350a0d6d059cad8ae03f8dadae and set watcher; region=test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae., state=RS_ZK_REGION_SPLITTING, servername=10.11.3.105,53794,1359756635266, createTime=1359756649235, payload.length=0
      2013-02-01 14:10:49,239 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(265): master:53791-0x13c97cfa9760000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/e9c665350a0d6d059cad8ae03f8dadae
      2013-02-01 14:10:49,240 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] zookeeper.ZKAssign(782): regionserver:53794-0x13c97cfa9760002 Successfully transitioned node e9c665350a0d6d059cad8ae03f8dadae from RS_ZK_REGION_SPLITTING to RS_ZK_REGION_SPLITTING
      2013-02-01 14:10:49,240 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] zookeeper.ZKAssign(720): regionserver:53794-0x13c97cfa9760002 Attempting to transition node e9c665350a0d6d059cad8ae03f8dadae from RS_ZK_REGION_SPLITTING to RS_ZK_REGION_SPLITTING
      2013-02-01 14:10:49,240 DEBUG [hbase-am-zkevent-worker-pool-2-thread-17] zookeeper.ZKUtil(1643): master:53791-0x13c97cfa9760000 Retrieved 94 byte(s) of data from znode /hbase/region-in-transition/e9c665350a0d6d059cad8ae03f8dadae and set watcher; region=test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae., state=RS_ZK_REGION_SPLITTING, servername=10.11.3.105,53794,1359756635266, createTime=1359756649238, payload.length=0
      2013-02-01 14:10:49,241 DEBUG [hbase-am-zkevent-worker-pool-2-thread-17] master.AssignmentManager(641): Handling transition=RS_ZK_REGION_SPLITTING, server=10.11.3.105,53794,1359756635266, region=e9c665350a0d6d059cad8ae03f8dadae, current state from region state map =null
      2013-02-01 14:10:49,241 INFO  [hbase-am-zkevent-worker-pool-2-thread-17] master.RegionStates(242): Region {NAME => 'test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae.', STARTKEY => '', ENDKEY => '', ENCODED => e9c665350a0d6d059cad8ae03f8dadae,} transitioned from {test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae. state=OPEN, ts=1359756647443, server=10.11.3.105,53794,1359756635266} to {test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae. state=SPLITTING, ts=1359756649241, server=10.11.3.105,53794,1359756635266}
      2013-02-01 14:10:49,241 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] zookeeper.ZKUtil(1643): regionserver:53794-0x13c97cfa9760002 Retrieved 94 byte(s) of data from znode /hbase/region-in-transition/e9c665350a0d6d059cad8ae03f8dadae; data=region=test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae., state=RS_ZK_REGION_SPLITTING, servername=10.11.3.105,53794,1359756635266, createTime=1359756649238, payload.length=0
      2013-02-01 14:10:49,243 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(265): master:53791-0x13c97cfa9760000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/e9c665350a0d6d059cad8ae03f8dadae
      2013-02-01 14:10:49,243 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] zookeeper.ZKAssign(782): regionserver:53794-0x13c97cfa9760002 Successfully transitioned node e9c665350a0d6d059cad8ae03f8dadae from RS_ZK_REGION_SPLITTING to RS_ZK_REGION_SPLITTING
      2013-02-01 14:10:49,244 DEBUG [hbase-am-zkevent-worker-pool-2-thread-18] zookeeper.ZKUtil(1643): master:53791-0x13c97cfa9760000 Retrieved 94 byte(s) of data from znode /hbase/region-in-transition/e9c665350a0d6d059cad8ae03f8dadae and set watcher; region=test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae., state=RS_ZK_REGION_SPLITTING, servername=10.11.3.105,53794,1359756635266, createTime=1359756649242, payload.length=0
      2013-02-01 14:10:49,245 DEBUG [hbase-am-zkevent-worker-pool-2-thread-18] master.AssignmentManager(641): Handling transition=RS_ZK_REGION_SPLITTING, server=10.11.3.105,53794,1359756635266, region=e9c665350a0d6d059cad8ae03f8dadae, current state from region state map =null
      2013-02-01 14:10:49,245 INFO  [hbase-am-zkevent-worker-pool-2-thread-18] master.RegionStates(242): Region {NAME => 'test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae.', STARTKEY => '', ENDKEY => '', ENCODED => e9c665350a0d6d059cad8ae03f8dadae,} transitioned from {test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae. state=SPLITTING, ts=1359756649241, server=10.11.3.105,53794,1359756635266} to {test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae. state=SPLITTING, ts=1359756649245, server=10.11.3.105,53794,1359756635266}
      2013-02-01 14:10:49,246 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] regionserver.HRegion(1005): Closing test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae.: disabling compactions & flushes
      2013-02-01 14:10:49,246 WARN  [IPC Server handler 1 on 53794] regionserver.HRegion(2133): Failed getting lock in batch put, row=hox
      org.apache.hadoop.hbase.NotServingRegionException: test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae. is closing
        at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:5319)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalObtainRowLock(HRegion.java:3181)
        at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:3269)
        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2130)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2004)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:3862)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3307)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.ProtobufRpcServerEngine$Server.call(ProtobufRpcServerEngine.java:197)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1786)
      2013-02-01 14:10:49,257 DEBUG [RegionServer:1;10.11.3.105,53794,1359756635266-splits-1359756649234] regionserver.HRegion(1027): Updates disabled for region test,,1359756646966.e9c665350a0d6d059cad8ae03f8dadae.
      

      I think we should replace IncreasingToUpperBoundRegionSplitPolicy in TestSnapshotFromMaster.

        Attachments

        1. 7739.txt
          2 kB
          Ted Yu

          Activity

            People

            • Assignee:
              yuzhihong@gmail.com Ted Yu
              Reporter:
              yuzhihong@gmail.com Ted Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: